subject:"\[openstack\-dev\] \[octavia\] how to recreate amphora instances"

Re: [openstack-dev] [octavia] how to recreate amphora instances

2017-11-13 Thread mihaela.balas

Hi Michael,

Thank you for the detailed explanation. I was in the worst scenario where the 
database entries were purged and I had to manually re-create the DB entries and 
the ports. I successfully managed to insert the lines in the database and the 
amphoras were created.

Thanks a lot for the hints. I also increased the DB cleanup interval to 1 week.

Mihaela

-Original Message-
From: Michael Johnson [mailto:johnso...@gmail.com] 
Sent: Friday, November 10, 2017 3:31 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [octavia] how to recreate amphora instances

I can give it a go, there are also logs of our conversation on evesdrop here: 
http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2017-11-02.log.html#t2017-11-02T11:07:45

Short background, they had an infrastructure issue where networking was out.  
This caused the load balancer amphora to be detected as failed and started the 
failover process which rebuilds the amphora, however this process also failed 
due to the wider infrastructure issues (Note, this is what I remember from the 
conversation. Correct the background if I am wrong). At this point the load 
balancers would be in a provisioning status of "ERROR", the amphora instances 
were likely deleted (depending on where the infrastructure issue impacted the 
failover process), and the amphora db records would be in the "DELETED" state.  
To make their situation worse, the DB cleanup cycle of the housekeeping process 
was set low (default is a week) and the amphora records were purged. This meant 
the customer data for the VIP address/ports was also purged.

Recovery:
If the database records were not purged, after the infrastructure issues are 
resolved, you can simply go into the octavia database and
issue: update load_balancer set provisioning_status="ACTIVE" where 
provisioning_status = "ERROR"; This will restart the health manager expecting 
the amphora to report health heartbeats and the failover process with start 
again rebuilding the amphora.  We have also recently added an admin API to 
trigger a failover manually 
(https://developer.openstack.org/api-ref/load-balancer/v2/index.html#failover-a-load-balancer).

If your amphora records have been purged, you are in some pain (don't do this). 
 This means that the VIP address and neutron port information for that VIP 
address are not available for the failover process to rebuild.  In this case, 
before you start the above process you will need to recreate the amphora 
records from your logs, either adding the port information if the ports are 
still live in neutron, or creating replacement ports.

Michael

On Wed, Nov 8, 2017 at 2:07 AM,  <mihaela.ba...@orange.com> wrote:
> I am also interested how to fix this. If you can describe shortly the 
> procedure.
>
> Thanks,
> Mihaela
>
> -Original Message-
> From: Michael Johnson [mailto:johnso...@gmail.com]
> Sent: Monday, November 06, 2017 6:54 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [octavia] how to recreate amphora 
> instances
>
> I think we helped you get going again in the IRC channel.  Please ping us 
> again in the IRC channel if you need more assistance.
>
> Michael
>
> On Thu, Nov 2, 2017 at 4:42 AM, Kim-Norman Sahm <kim-norman.s...@noris.de> 
> wrote:
>> Hi,
>>
>> after a rabbitmq problem octavia has removed all amphora instances.
>> the loadbalancers are in provisioning_status "ACTIVE"
>>
>> ~$ neutron lbaas-loadbalancer-list
>> neutron CLI is deprecated and will be removed in the future. Use 
>> openstack CLI instead.
>> | 07b41df6-bb75-4502-975a-20140b0832dd | Load Balancer
>> 4   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.18   | ACTIVE  | octavia  |
>> | 25664be7-15cb-426b-ad09-6102afb62b14 | Load Balancer
>> 2   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.7| ACTIVE  | octavia  |
>> | 927eb754-7c52-4060-b130-1f5e82d92555 | Load Balancer
>> 6   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.17   | ACTIVE  | octavia  |
>> | b4d93c68-89d6-4e4f-b06c-117d4ea933fa | Load Balancer
>> 5   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.24   | ACTIVE  | octavia  |
>> | d7699f8d-2106-42d6-8797-5feb72de6e2e | Load Balancer
>> 1   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.5| ACTIVE  | octavia  |
>> | dd6114ae-21e9-41bd-b155-325287aed420 | Load Balancer
>> 3   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.23   | ACTIVE  | octavia  |
>>
>> How can we trigge

Re: [openstack-dev] [octavia] how to recreate amphora instances

2017-11-09 Thread Michael Johnson

I can give it a go, there are also logs of our conversation on
evesdrop here: 
http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2017-11-02.log.html#t2017-11-02T11:07:45

Short background, they had an infrastructure issue where networking
was out.  This caused the load balancer amphora to be detected as
failed and started the failover process which rebuilds the amphora,
however this process also failed due to the wider infrastructure
issues (Note, this is what I remember from the conversation. Correct
the background if I am wrong). At this point the load balancers would
be in a provisioning status of "ERROR", the amphora instances were
likely deleted (depending on where the infrastructure issue impacted
the failover process), and the amphora db records would be in the
"DELETED" state.  To make their situation worse, the DB cleanup cycle
of the housekeeping process was set low (default is a week) and the
amphora records were purged. This meant the customer data for the VIP
address/ports was also purged.

Recovery:
If the database records were not purged, after the infrastructure
issues are resolved, you can simply go into the octavia database and
issue: update load_balancer set provisioning_status="ACTIVE" where
provisioning_status = "ERROR";
This will restart the health manager expecting the amphora to report
health heartbeats and the failover process with start again rebuilding
the amphora.  We have also recently added an admin API to trigger a
failover manually
(https://developer.openstack.org/api-ref/load-balancer/v2/index.html#failover-a-load-balancer).

If your amphora records have been purged, you are in some pain (don't
do this).  This means that the VIP address and neutron port
information for that VIP address are not available for the failover
process to rebuild.  In this case, before you start the above process
you will need to recreate the amphora records from your logs, either
adding the port information if the ports are still live in neutron, or
creating replacement ports.

Michael

On Wed, Nov 8, 2017 at 2:07 AM,  <mihaela.ba...@orange.com> wrote:
> I am also interested how to fix this. If you can describe shortly the 
> procedure.
>
> Thanks,
> Mihaela
>
> -Original Message-
> From: Michael Johnson [mailto:johnso...@gmail.com]
> Sent: Monday, November 06, 2017 6:54 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [octavia] how to recreate amphora instances
>
> I think we helped you get going again in the IRC channel.  Please ping us 
> again in the IRC channel if you need more assistance.
>
> Michael
>
> On Thu, Nov 2, 2017 at 4:42 AM, Kim-Norman Sahm <kim-norman.s...@noris.de> 
> wrote:
>> Hi,
>>
>> after a rabbitmq problem octavia has removed all amphora instances.
>> the loadbalancers are in provisioning_status "ACTIVE"
>>
>> ~$ neutron lbaas-loadbalancer-list
>> neutron CLI is deprecated and will be removed in the future. Use
>> openstack CLI instead.
>> | 07b41df6-bb75-4502-975a-20140b0832dd | Load Balancer
>> 4   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.18   | ACTIVE  | octavia  |
>> | 25664be7-15cb-426b-ad09-6102afb62b14 | Load Balancer
>> 2   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.7| ACTIVE  | octavia  |
>> | 927eb754-7c52-4060-b130-1f5e82d92555 | Load Balancer
>> 6   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.17   | ACTIVE  | octavia  |
>> | b4d93c68-89d6-4e4f-b06c-117d4ea933fa | Load Balancer
>> 5   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.24   | ACTIVE  | octavia  |
>> | d7699f8d-2106-42d6-8797-5feb72de6e2e | Load Balancer
>> 1   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.5| ACTIVE  | octavia  |
>> | dd6114ae-21e9-41bd-b155-325287aed420 | Load Balancer
>> 3   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.23   | ACTIVE  | octavia  |
>>
>> How can we trigger octavia to rebuild the amphore instances?
>> I've tried to restart the octavia services but it didn't solved the
>> problem.
>>
>> Best regards
>> Kim
>>
>>
>> Kim-Norman Sahm
>> Cloud & Infrastructure(OCI)
>> noris network AG
>> Thomas-Mann-Straße 16-20
>> 90471 Nürnberg
>> Deutschland
>> Tel +49 911 9352 1433
>> Fax +49 911 9352 100
>>
>> kim-norman.s...@noris.de
>> https://www.noris.de - Mehr Leistung als Standard
>> Vorstand: Ingo Kraupa (Vorsit

Re: [openstack-dev] [octavia] how to recreate amphora instances

2017-11-08 Thread mihaela.balas

I am also interested how to fix this. If you can describe shortly the procedure.

Thanks,
Mihaela

-Original Message-
From: Michael Johnson [mailto:johnso...@gmail.com] 
Sent: Monday, November 06, 2017 6:54 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [octavia] how to recreate amphora instances

I think we helped you get going again in the IRC channel.  Please ping us again 
in the IRC channel if you need more assistance.

Michael

On Thu, Nov 2, 2017 at 4:42 AM, Kim-Norman Sahm <kim-norman.s...@noris.de> 
wrote:
> Hi,
>
> after a rabbitmq problem octavia has removed all amphora instances.
> the loadbalancers are in provisioning_status "ACTIVE"
>
> ~$ neutron lbaas-loadbalancer-list
> neutron CLI is deprecated and will be removed in the future. Use 
> openstack CLI instead.
> | 07b41df6-bb75-4502-975a-20140b0832dd | Load Balancer
> 4   | 260b0c452e214accaf6cc0e98fb10fc0 |
> 192.168.1.18   | ACTIVE  | octavia  |
> | 25664be7-15cb-426b-ad09-6102afb62b14 | Load Balancer
> 2   | 260b0c452e214accaf6cc0e98fb10fc0 |
> 192.168.1.7| ACTIVE  | octavia  |
> | 927eb754-7c52-4060-b130-1f5e82d92555 | Load Balancer
> 6   | 260b0c452e214accaf6cc0e98fb10fc0 |
> 192.168.1.17   | ACTIVE  | octavia  |
> | b4d93c68-89d6-4e4f-b06c-117d4ea933fa | Load Balancer
> 5   | 260b0c452e214accaf6cc0e98fb10fc0 |
> 192.168.1.24   | ACTIVE  | octavia  |
> | d7699f8d-2106-42d6-8797-5feb72de6e2e | Load Balancer
> 1   | 260b0c452e214accaf6cc0e98fb10fc0 |
> 192.168.1.5| ACTIVE  | octavia  |
> | dd6114ae-21e9-41bd-b155-325287aed420 | Load Balancer
> 3   | 260b0c452e214accaf6cc0e98fb10fc0 |
> 192.168.1.23   | ACTIVE  | octavia  |
>
> How can we trigger octavia to rebuild the amphore instances?
> I've tried to restart the octavia services but it didn't solved the 
> problem.
>
> Best regards
> Kim
>
>
> Kim-Norman Sahm
> Cloud & Infrastructure(OCI)
> noris network AG
> Thomas-Mann-Straße 16-20
> 90471 Nürnberg
> Deutschland
> Tel +49 911 9352 1433
> Fax +49 911 9352 100
>
> kim-norman.s...@noris.de
> https://www.noris.de - Mehr Leistung als Standard
> Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel Vorsitzender des 
> Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689
>
>
>
>
>
> __
>  OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [octavia] how to recreate amphora instances

2017-11-06 Thread Michael Johnson

I think we helped you get going again in the IRC channel.  Please ping
us again in the IRC channel if you need more assistance.

Michael

On Thu, Nov 2, 2017 at 4:42 AM, Kim-Norman Sahm
 wrote:
> Hi,
>
> after a rabbitmq problem octavia has removed all amphora instances.
> the loadbalancers are in provisioning_status "ACTIVE"
>
> ~$ neutron lbaas-loadbalancer-list
> neutron CLI is deprecated and will be removed in the future. Use
> openstack CLI instead.
> | 07b41df6-bb75-4502-975a-20140b0832dd | Load Balancer
> 4   | 260b0c452e214accaf6cc0e98fb10fc0 |
> 192.168.1.18   | ACTIVE  | octavia  |
> | 25664be7-15cb-426b-ad09-6102afb62b14 | Load Balancer
> 2   | 260b0c452e214accaf6cc0e98fb10fc0 |
> 192.168.1.7| ACTIVE  | octavia  |
> | 927eb754-7c52-4060-b130-1f5e82d92555 | Load Balancer
> 6   | 260b0c452e214accaf6cc0e98fb10fc0 |
> 192.168.1.17   | ACTIVE  | octavia  |
> | b4d93c68-89d6-4e4f-b06c-117d4ea933fa | Load Balancer
> 5   | 260b0c452e214accaf6cc0e98fb10fc0 |
> 192.168.1.24   | ACTIVE  | octavia  |
> | d7699f8d-2106-42d6-8797-5feb72de6e2e | Load Balancer
> 1   | 260b0c452e214accaf6cc0e98fb10fc0 |
> 192.168.1.5| ACTIVE  | octavia  |
> | dd6114ae-21e9-41bd-b155-325287aed420 | Load Balancer
> 3   | 260b0c452e214accaf6cc0e98fb10fc0 |
> 192.168.1.23   | ACTIVE  | octavia  |
>
> How can we trigger octavia to rebuild the amphore instances?
> I've tried to restart the octavia services but it didn't solved the
> problem.
>
> Best regards
> Kim
>
>
> Kim-Norman Sahm
> Cloud & Infrastructure(OCI)
> noris network AG
> Thomas-Mann-Straße 16-20
> 90471 Nürnberg
> Deutschland
> Tel +49 911 9352 1433
> Fax +49 911 9352 100
>
> kim-norman.s...@noris.de
> https://www.noris.de - Mehr Leistung als Standard
> Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel
> Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689
>
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [octavia] how to recreate amphora instances

2017-11-02 Thread Kim-Norman Sahm

Hi,

after a rabbitmq problem octavia has removed all amphora instances.
the loadbalancers are in provisioning_status "ACTIVE"

~$ neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use
openstack CLI instead.
| 07b41df6-bb75-4502-975a-20140b0832dd | Load Balancer
4   | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.18   | ACTIVE  | octavia  |
| 25664be7-15cb-426b-ad09-6102afb62b14 | Load Balancer
2   | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.7| ACTIVE  | octavia  |
| 927eb754-7c52-4060-b130-1f5e82d92555 | Load Balancer
6   | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.17   | ACTIVE  | octavia  |
| b4d93c68-89d6-4e4f-b06c-117d4ea933fa | Load Balancer
5   | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.24   | ACTIVE  | octavia  |
| d7699f8d-2106-42d6-8797-5feb72de6e2e | Load Balancer
1   | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.5| ACTIVE  | octavia  |
| dd6114ae-21e9-41bd-b155-325287aed420 | Load Balancer
3   | 260b0c452e214accaf6cc0e98fb10fc0 |
192.168.1.23   | ACTIVE  | octavia  |

How can we trigger octavia to rebuild the amphore instances?
I've tried to restart the octavia services but it didn't solved the
problem.

Best regards
Kim


Kim-Norman Sahm
Cloud & Infrastructure(OCI)
noris network AG
Thomas-Mann-Straße 16-20
90471 Nürnberg
Deutschland
Tel +49 911 9352 1433
Fax +49 911 9352 100

kim-norman.s...@noris.de
https://www.noris.de - Mehr Leistung als Standard
Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689






smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [octavia] how to recreate amphora instances

Re: [openstack-dev] [octavia] how to recreate amphora instances

Re: [openstack-dev] [octavia] how to recreate amphora instances

Re: [openstack-dev] [octavia] how to recreate amphora instances

[openstack-dev] [octavia] how to recreate amphora instances

5 matches

Site Navigation

Mail list logo

Footer information