Re: [openstack-dev] [Neutron] AZ Support

[Takashi Yamamoto]

On Mon, Oct 5, 2015 at 10:41 PM, Ihar Hrachyshka  wrote:
>> On 05 Oct 2015, at 15:32, Gary Kotton  wrote:
>>
>>
>>
>> On 10/5/15, 3:21 AM, "Ihar Hrachyshka"  wrote:
>>
 On 04 Oct 2015, at 19:21, Gary Kotton  wrote:

 Sorry, it is not a result of the AZ support (humble apologies)

 It is a result of https://review.openstack.org/#/c/226362/
>>>
>>> So you use DHCP agent with non-ml2 plugin, and it broke you. Do you think
>>> it¹s ok for you to change the RPC topic to work with Mitaka, or we¹ll
>>> need to handle it more gracefully, f.e. by detecting the reply timeout
>>> and switching back to the old topic?
>>
>> My thinking is that we should fail to use the old topic. But then again, I
>> would expect that the Neutron service would first be upgraded and then the
>> agents. Updating the agents first would be a recipe for disaster.
>
> Yes, controller first, then agents. That’s why there was no fallback 
> mechanism in that patch that broke you, while we looked into backwards 
> compatibility. Though no one considered the case of 3party plugins that may 
> rely on some in-tree agents. We can accommodate for them, if it seems too 
> much effort for them to change the topic name they report on. But note that 
> it would also mean that those plugins don’t utilize the separate threads to 
> handle reports, which can be bad for their scalability.

i made a fix for midonet.  just FYI.
https://review.openstack.org/#/c/230333/

>
> Ihar
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] AZ Support

[Gary Kotton]

Hi,
Yes, you are correct. That patch is the culprit (my bad and once again humble 
apologies)

Regarding the AZ support I think that we need to do the following:

  1.  Have this in a separate topic until it is complete. I have a number of 
concerns here:
 *   The upgrade impact on Nova. Today in Nova one can have N AZ’s and they 
will all work on the same virtual networks. It is not clear how this will work 
here and if the networks can be shared across AZ’s (maybe this was discussed 
and I am missing somehing)
 *   A few weeks ago Monty raised concerns about floating IP support. I 
think that this will be required for AZ support. In the past one could have a 
shared network between AZ’s and now they will need two isolated networks
  2.  In Nova on of the top priority features over the last few cycles has been 
cells. At the moment there is no cell support for Neutron. I feel that the AZ 
support is someone what related and maybe we should try and kill 2 birds with 
one stone here and have the cell support combined if possible. I think that 
this is certainly something that should be a cross project summit discussion.

Thanks
Gary

From: "Armando M." <arma...@gmail.com<mailto:arma...@gmail.com>>
Reply-To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Sunday, October 4, 2015 at 8:18 PM
To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Neutron] AZ Support



On 4 October 2015 at 10:00, Gary Kotton 
<gkot...@vmware.com<mailto:gkot...@vmware.com>> wrote:
The DHCP agent has the following exception:

2015-10-02 23:57:05.787 ERROR neutron.agent.dhcp.agent 
[req-17c3aa12-41fd-41f8-8e23-2f9740e50746 None None] Failed reporting state!
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent Traceback (most recent 
call last):
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 572, in _report_state
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent 
self.state_rpc.report_state(ctx, self.agent_state, self.use_call)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/opt/stack/neutron/neutron/agent/rpc.py", line 86, in report_state
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent return 
method(context, 'report_state', **kwargs)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 
158, in call
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent retry=self.retry)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, 
in _send
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent timeout=timeout, 
retry=retry)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 431, in send
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent retry=retry)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 420, in _send
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent result = 
self._waiter.wait(msg_id, timeout)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 318, in wait
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent message = 
self.waiters.get(msg_id, timeout=timeout)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 223, in get
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent 'to message ID %s' % 
msg_id)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent MessagingTimeout: Timed 
out waiting for a reply to message ID f4ed0bd26feb462c9b7b49a6d85caeae

Now when I use the stable/liberty branch everything is OK


Ah, I suspect that's your culprit:

https://review.openstack.org/#/c/226362/

instead of AZ's initial support.

Thanks
Gary

From: "Armando M." <arma...@gmail.com<mailto:arma...@gmail.com>>
Reply-To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Sunday, October 4, 2015 at 7:34 PM
To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Neutron] AZ Support



On 4 October 2015 at 09:19, Gary Kotton 
<gkot...@vmware.com<mailto:gkot...@vmware.com>> wrote:
Hi,
It appears that the addition has broken the vmware_nsx plugin 
(https://review.openstack.org/183369). We are 

Re: [openstack-dev] [Neutron] AZ Support

[Hirofumi Ichihara]

Hi Gary,

Thank you for your suggestion.
In Liberty cycle[1], I have discussed about these points with neutron folks and 
network operators.


> On 2015/10/05, at 15:54, Gary Kotton <gkot...@vmware.com> wrote:
> 
> Hi,
> Yes, you are correct. That patch is the culprit (my bad and once again humble 
> apologies)
> 
> Regarding the AZ support I think that we need to do the following:
> Have this in a separate topic until it is complete. I have a number of 
> concerns here:
> The upgrade impact on Nova. Today in Nova one can have N AZ’s and they will 
> all work on the same virtual networks. It is not clear how this will work 
> here and if the networks can be shared across AZ’s (maybe this was discussed 
> and I am missing somehing)
> A few weeks ago Monty raised concerns about floating IP support. I think that 
> this will be required for AZ support. In the past one could have a shared 
> network between AZ’s and now they will need two isolated networks
I guess that this is similar to “Segment” discussion[2].
I tried to include this point to Availability Zone spec so that we can deploy 
an environment has network unreachable AZs.
However, some folks disagreed with this because availability zone must be 
something to give users High Availability not manage isolated network with AZ.

> 
> In Nova on of the top priority features over the last few cycles has been 
> cells. At the moment there is no cell support for Neutron. I feel that the AZ 
> support is someone what related and maybe we should try and kill 2 birds with 
> one stone here and have the cell support combined if possible. I think that 
> this is certainly something that should be a cross project summit discussion.
I think too that Neutron Cell Support is important. However, neutron quite 
depends on the backend unlike nova.
It’s hard to combine it and it will take a long time. AZ support is also 
different from Cell support.
We should not discuss these to connect Neutron AZ support with Cell support 
because I’m afraid to fall between two stools.
I agree that we have a cross project summit discussion to connect Nova Cell 
support with Neutron Cell support.

[1]: https://review.openstack.org/#/c/169612/ 
<https://review.openstack.org/#/c/169612/>
[2]: https://bugs.launchpad.net/neutron/+bug/1458890 
<https://bugs.launchpad.net/neutron/+bug/1458890>

Thanks,
Hirofumi


> Thanks
> Gary
> 
> From: "Armando M." <arma...@gmail.com <mailto:arma...@gmail.com>>
> Reply-To: OpenStack List <openstack-dev@lists.openstack.org 
> <mailto:openstack-dev@lists.openstack.org>>
> Date: Sunday, October 4, 2015 at 8:18 PM
> To: OpenStack List <openstack-dev@lists.openstack.org 
> <mailto:openstack-dev@lists.openstack.org>>
> Subject: Re: [openstack-dev] [Neutron] AZ Support
> 
> 
> 
> On 4 October 2015 at 10:00, Gary Kotton <gkot...@vmware.com 
> <mailto:gkot...@vmware.com>> wrote:
>> The DHCP agent has the following exception:
>> 
>> 2015-10-02 23:57:05.787 ERROR neutron.agent.dhcp.agent 
>> [req-17c3aa12-41fd-41f8-8e23-2f9740e50746 None None] Failed reporting state!
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent Traceback (most 
>> recent call last):
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
>> "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 572, in _report_state
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent 
>> self.state_rpc.report_state(ctx, self.agent_state, self.use_call)
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
>> "/opt/stack/neutron/neutron/agent/rpc.py", line 86, in report_state
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent return 
>> method(context, 'report_state', **kwargs)
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
>> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 
>> 158, in call
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent retry=self.retry)
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
>> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 
>> 90, in _send
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent timeout=timeout, 
>> retry=retry)
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
>> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
>>  line 431, in send
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent retry=retry)
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
>> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.p

Re: [openstack-dev] [Neutron] AZ Support

[Ihar Hrachyshka]

> On 04 Oct 2015, at 19:21, Gary Kotton  wrote:
> 
> Sorry, it is not a result of the AZ support (humble apologies)
> 
> It is a result of https://review.openstack.org/#/c/226362/

So you use DHCP agent with non-ml2 plugin, and it broke you. Do you think it’s 
ok for you to change the RPC topic to work with Mitaka, or we’ll need to handle 
it more gracefully, f.e. by detecting the reply timeout and switching back to 
the old topic?

Ihar


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] AZ Support

[Akihiro Motoki]

Hi,

2015-10-05 15:54 GMT+09:00 Gary Kotton <gkot...@vmware.com>:

> Hi,
> Yes, you are correct. That patch is the culprit (my bad and once again
> humble apologies)
>
> Regarding the AZ support I think that we need to do the following:
>
>1. Have this in a separate topic until it is complete. I have a number
>of concerns here:
>   1. The upgrade impact on Nova. Today in Nova one can have N AZ’s
>   and they will all work on the same virtual networks. It is not clear how
>   this will work here and if the networks can be shared across AZ’s (maybe
>   this was discussed and I am missing somehing)
>
> The proposed "AZ support" feature in Neutron is the really initial step
and we have many points which can be improved.
The first step of AZ support is only related to the agent scheduling andit
just tries to assign agent(s) from AZ(s) which users want.
Networks are still shared across AZs, so I don't think there are upgrade
impacts on Nova.


>1.
>   1. A few weeks ago Monty raised concerns about floating IP support.
>   I think that this will be required for AZ support. In the past one could
>   have a shared network between AZ’s and now they will need two isolated
>   networks
>
> I think Floating IP pool per AZ is a straight forward approach.
However, this does not necessarily mean an external network should limit to
a specific AZ.
Of course, it is better that VMs in AZ1 connect to a network (whose
dhcp-agent lives in AZ1)
and the network is connected to router in AZ1.
The question is how users can select floating IP pool but in this case
users can know which FIP pool belongs to which AZ.

There are other approaches. One possible option is to have one external
network and
a router supports multiple upstream links using routing protocols.

I just wrote approaches in my mind and there must be more options.


>1. In Nova on of the top priority features over the last few cycles
>has been cells. At the moment there is no cell support for Neutron. I feel
>that the AZ support is someone what related and maybe we should try and
>kill 2 birds with one stone here and have the cell support combined if
>possible. I think that this is certainly something that should be a cross
>project summit discussion.
>
>
I agree that Cells support is one of missing areas in Neutron.
I think AZ support and Cells support are different things though they are
similar.
AZ support is visible to users and Cells are not.

Thanks,
Akihiro



> Thanks
> Gary
>
> From: "Armando M." <arma...@gmail.com>
> Reply-To: OpenStack List <openstack-dev@lists.openstack.org>
> Date: Sunday, October 4, 2015 at 8:18 PM
>
> To: OpenStack List <openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [Neutron] AZ Support
>
>
>
> On 4 October 2015 at 10:00, Gary Kotton <gkot...@vmware.com> wrote:
>
>> The DHCP agent has the following exception:
>>
>> 2015-10-02 23:57:05.787 ERROR neutron.agent.dhcp.agent
>> [req-17c3aa12-41fd-41f8-8e23-2f9740e50746 None None] Failed reporting state!
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent Traceback (most
>> recent call last):
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
>> "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 572, in _report_state
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent
>> self.state_rpc.report_state(ctx, self.agent_state, self.use_call)
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
>> "/opt/stack/neutron/neutron/agent/rpc.py", line 86, in report_state
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent return
>> method(context, 'report_state', **kwargs)
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
>> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line
>> 158, in call
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent
>> retry=self.retry)
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
>> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line
>> 90, in _send
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent
>> timeout=timeout, retry=retry)
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
>> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
>> line 431, in send
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent retry=retry)
>> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
>> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
>> line 420

Re: [openstack-dev] [Neutron] AZ Support

[Gary Kotton]

On 10/5/15, 3:21 AM, "Ihar Hrachyshka"  wrote:

>> On 04 Oct 2015, at 19:21, Gary Kotton  wrote:
>> 
>> Sorry, it is not a result of the AZ support (humble apologies)
>> 
>> It is a result of https://review.openstack.org/#/c/226362/
>
>So you use DHCP agent with non-ml2 plugin, and it broke you. Do you think
>it¹s ok for you to change the RPC topic to work with Mitaka, or we¹ll
>need to handle it more gracefully, f.e. by detecting the reply timeout
>and switching back to the old topic?

My thinking is that we should fail to use the old topic. But then again, I
would expect that the Neutron service would first be upgraded and then the
agents. Updating the agents first would be a recipe for disaster.

>
>Ihar


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] AZ Support

[Gary Kotton]

From: Akihiro MOTOKI <amot...@gmail.com<mailto:amot...@gmail.com>>
Reply-To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Monday, October 5, 2015 at 3:48 PM
To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Neutron] AZ Support

Hi,

2015-10-05 15:54 GMT+09:00 Gary Kotton 
<gkot...@vmware.com<mailto:gkot...@vmware.com>>:
Hi,
Yes, you are correct. That patch is the culprit (my bad and once again humble 
apologies)

Regarding the AZ support I think that we need to do the following:

  1.  Have this in a separate topic until it is complete. I have a number of 
concerns here:
 *   The upgrade impact on Nova. Today in Nova one can have N AZ’s and they 
will all work on the same virtual networks. It is not clear how this will work 
here and if the networks can be shared across AZ’s (maybe this was discussed 
and I am missing somehing)

The proposed "AZ support" feature in Neutron is the really initial step and we 
have many points which can be improved.
The first step of AZ support is only related to the agent scheduling andit just 
tries to assign agent(s) from AZ(s) which users want.
Networks are still shared across AZs, so I don't think there are upgrade 
impacts on Nova.

[Gary] Thanks for that clarification. From 
https://review.openstack.org/#/c/204436/ is did not really seem like that is 
the case, but I need to go over that in more detail. When will Nova know when 
to pass the AZ to Neutron? Will this be if the extension is supported? I am 
just failing to understand  the nova side of the integrations.


  1.
 *   A few weeks ago Monty raised concerns about floating IP support. I 
think that this will be required for AZ support. In the past one could have a 
shared network between AZ’s and now they will need two isolated networks

I think Floating IP pool per AZ is a straight forward approach.
However, this does not necessarily mean an external network should limit to a 
specific AZ.
Of course, it is better that VMs in AZ1 connect to a network (whose dhcp-agent 
lives in AZ1)

[Gary] So each time a new AZ is created the admin will need to update the 
Neutron configuration file - 
https://review.openstack.org/#/c/183369/40/etc/neutron.conf,cm?

and the network is connected to router in AZ1.
The question is how users can select floating IP pool but in this case users 
can know which FIP pool belongs to which AZ.

There are other approaches. One possible option is to have one external network 
and
a router supports multiple upstream links using routing protocols.

I just wrote approaches in my mind and there must be more options.


  1.  In Nova on of the top priority features over the last few cycles has been 
cells. At the moment there is no cell support for Neutron. I feel that the AZ 
support is someone what related and maybe we should try and kill 2 birds with 
one stone here and have the cell support combined if possible. I think that 
this is certainly something that should be a cross project summit discussion.

I agree that Cells support is one of missing areas in Neutron.
I think AZ support and Cells support are different things though they are 
similar.
AZ support is visible to users and Cells are not.

Thanks,
Akihiro


Thanks
Gary

From: "Armando M." <arma...@gmail.com<mailto:arma...@gmail.com>>
Reply-To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Sunday, October 4, 2015 at 8:18 PM

To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Neutron] AZ Support



On 4 October 2015 at 10:00, Gary Kotton 
<gkot...@vmware.com<mailto:gkot...@vmware.com>> wrote:
The DHCP agent has the following exception:

2015-10-02 23:57:05.787 ERROR neutron.agent.dhcp.agent 
[req-17c3aa12-41fd-41f8-8e23-2f9740e50746 None None] Failed reporting state!
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent Traceback (most recent 
call last):
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 572, in _report_state
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent 
self.state_rpc.report_state(ctx, self.agent_state, self.use_call)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/opt/stack/neutron/neutron/agent/rpc.py", line 86, in report_state
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent return 
method(context, 'report_state', **kwargs)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 
158, in call
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent retry=self.retry)
2015-10-02 23:57:05.787 TRACE neutr

Re: [openstack-dev] [Neutron] AZ Support

[Ihar Hrachyshka]

> On 05 Oct 2015, at 15:32, Gary Kotton  wrote:
> 
> 
> 
> On 10/5/15, 3:21 AM, "Ihar Hrachyshka"  wrote:
> 
>>> On 04 Oct 2015, at 19:21, Gary Kotton  wrote:
>>> 
>>> Sorry, it is not a result of the AZ support (humble apologies)
>>> 
>>> It is a result of https://review.openstack.org/#/c/226362/
>> 
>> So you use DHCP agent with non-ml2 plugin, and it broke you. Do you think
>> it¹s ok for you to change the RPC topic to work with Mitaka, or we¹ll
>> need to handle it more gracefully, f.e. by detecting the reply timeout
>> and switching back to the old topic?
> 
> My thinking is that we should fail to use the old topic. But then again, I
> would expect that the Neutron service would first be upgraded and then the
> agents. Updating the agents first would be a recipe for disaster.

Yes, controller first, then agents. That’s why there was no fallback mechanism 
in that patch that broke you, while we looked into backwards compatibility. 
Though no one considered the case of 3party plugins that may rely on some 
in-tree agents. We can accommodate for them, if it seems too much effort for 
them to change the topic name they report on. But note that it would also mean 
that those plugins don’t utilize the separate threads to handle reports, which 
can be bad for their scalability.

Ihar


signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] AZ Support

[Armando M.]

On 4 October 2015 at 09:19, Gary Kotton  wrote:

> Hi,
> It appears that the addition has broken the vmware_nsx plugin (
> https://review.openstack.org/183369). We are still debugging. Would it be
> worthwhile considering adding this support as a feature branch and then
> when the entire feature is ready that we merge it to the tree. This will
> enable the external vendors to be alive and kicking.
>

Please let us know how we can help you to fix it. The CI hasn't voted on
this patch since May 14, so clearly the breakage flew under the radar.


> Thanks
> Gary
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] AZ Support

[Armando M.]

On 4 October 2015 at 10:00, Gary Kotton <gkot...@vmware.com> wrote:

> The DHCP agent has the following exception:
>
> 2015-10-02 23:57:05.787 ERROR neutron.agent.dhcp.agent
> [req-17c3aa12-41fd-41f8-8e23-2f9740e50746 None None] Failed reporting state!
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent Traceback (most
> recent call last):
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
> "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 572, in _report_state
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent
> self.state_rpc.report_state(ctx, self.agent_state, self.use_call)
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
> "/opt/stack/neutron/neutron/agent/rpc.py", line 86, in report_state
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent return
> method(context, 'report_state', **kwargs)
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line
> 158, in call
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent
> retry=self.retry)
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line
> 90, in _send
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent
> timeout=timeout, retry=retry)
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 431, in send
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent retry=retry)
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 420, in _send
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent result =
> self._waiter.wait(msg_id, timeout)
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 318, in wait
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent message =
> self.waiters.get(msg_id, timeout=timeout)
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 223, in get
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent 'to message ID
> %s' % msg_id)
> 2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent MessagingTimeout:
> Timed out waiting for a reply to message ID f4ed0bd26feb462c9b7b49a6d85caeae
>
> Now when I use the stable/liberty branch everything is OK
>
>
Ah, I suspect that's your culprit:

https://review.openstack.org/#/c/226362/

instead of AZ's initial support.


> Thanks
> Gary
>
> From: "Armando M." <arma...@gmail.com>
> Reply-To: OpenStack List <openstack-dev@lists.openstack.org>
> Date: Sunday, October 4, 2015 at 7:34 PM
> To: OpenStack List <openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [Neutron] AZ Support
>
>
>
> On 4 October 2015 at 09:19, Gary Kotton <gkot...@vmware.com> wrote:
>
>> Hi,
>> It appears that the addition has broken the vmware_nsx plugin (
>> https://review.openstack.org/183369). We are still debugging. Would it
>> be worthwhile considering adding this support as a feature branch and then
>> when the entire feature is ready that we merge it to the tree. This will
>> enable the external vendors to be alive and kicking.
>>
>
> Please let us know how we can help you to fix it. The CI hasn't voted on
> this patch since May 14, so clearly the breakage flew under the radar.
>
>
>> Thanks
>> Gary
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] AZ Support

[Gary Kotton]

The DHCP agent has the following exception:

2015-10-02 23:57:05.787 ERROR neutron.agent.dhcp.agent 
[req-17c3aa12-41fd-41f8-8e23-2f9740e50746 None None] Failed reporting state!
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent Traceback (most recent 
call last):
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 572, in _report_state
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent 
self.state_rpc.report_state(ctx, self.agent_state, self.use_call)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/opt/stack/neutron/neutron/agent/rpc.py", line 86, in report_state
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent return 
method(context, 'report_state', **kwargs)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 
158, in call
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent retry=self.retry)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, 
in _send
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent timeout=timeout, 
retry=retry)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 431, in send
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent retry=retry)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 420, in _send
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent result = 
self._waiter.wait(msg_id, timeout)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 318, in wait
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent message = 
self.waiters.get(msg_id, timeout=timeout)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 223, in get
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent 'to message ID %s' % 
msg_id)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent MessagingTimeout: Timed 
out waiting for a reply to message ID f4ed0bd26feb462c9b7b49a6d85caeae

Now when I use the stable/liberty branch everything is OK

Thanks
Gary

From: "Armando M." <arma...@gmail.com<mailto:arma...@gmail.com>>
Reply-To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Sunday, October 4, 2015 at 7:34 PM
To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Neutron] AZ Support



On 4 October 2015 at 09:19, Gary Kotton 
<gkot...@vmware.com<mailto:gkot...@vmware.com>> wrote:
Hi,
It appears that the addition has broken the vmware_nsx plugin 
(https://review.openstack.org/183369). We are still debugging. Would it be 
worthwhile considering adding this support as a feature branch and then when 
the entire feature is ready that we merge it to the tree. This will enable the 
external vendors to be alive and kicking.

Please let us know how we can help you to fix it. The CI hasn't voted on this 
patch since May 14, so clearly the breakage flew under the radar.

Thanks
Gary

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] AZ Support

[Gary Kotton]

Sorry, it is not a result of the AZ support (humble apologies)

It is a result of https://review.openstack.org/#/c/226362/

From: Gary Kotton <gkot...@vmware.com<mailto:gkot...@vmware.com>>
Reply-To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Sunday, October 4, 2015 at 8:00 PM
To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Neutron] AZ Support

The DHCP agent has the following exception:

2015-10-02 23:57:05.787 ERROR neutron.agent.dhcp.agent 
[req-17c3aa12-41fd-41f8-8e23-2f9740e50746 None None] Failed reporting state!
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent Traceback (most recent 
call last):
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 572, in _report_state
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent 
self.state_rpc.report_state(ctx, self.agent_state, self.use_call)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/opt/stack/neutron/neutron/agent/rpc.py", line 86, in report_state
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent return 
method(context, 'report_state', **kwargs)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 
158, in call
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent retry=self.retry)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, 
in _send
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent timeout=timeout, 
retry=retry)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 431, in send
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent retry=retry)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 420, in _send
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent result = 
self._waiter.wait(msg_id, timeout)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 318, in wait
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent message = 
self.waiters.get(msg_id, timeout=timeout)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent   File 
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", 
line 223, in get
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent 'to message ID %s' % 
msg_id)
2015-10-02 23:57:05.787 TRACE neutron.agent.dhcp.agent MessagingTimeout: Timed 
out waiting for a reply to message ID f4ed0bd26feb462c9b7b49a6d85caeae

Now when I use the stable/liberty branch everything is OK

Thanks
Gary

From: "Armando M." <arma...@gmail.com<mailto:arma...@gmail.com>>
Reply-To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Sunday, October 4, 2015 at 7:34 PM
To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Neutron] AZ Support



On 4 October 2015 at 09:19, Gary Kotton 
<gkot...@vmware.com<mailto:gkot...@vmware.com>> wrote:
Hi,
It appears that the addition has broken the vmware_nsx plugin 
(https://review.openstack.org/183369). We are still debugging. Would it be 
worthwhile considering adding this support as a feature branch and then when 
the entire feature is ready that we merge it to the tree. This will enable the 
external vendors to be alive and kicking.

Please let us know how we can help you to fix it. The CI hasn't voted on this 
patch since May 14, so clearly the breakage flew under the radar.

Thanks
Gary

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev