Re: [openstack-dev] [neutron][taas] neutron ovs-agent deletes taas flows

[SUZUKI, Kazuhiro]

Hi,

I submited a modified patch set as follows.
In addition, I added test codes for this patch.

>   1. Set an integer cookie value to taas flows.
>  Maybe all "1" for short term tentative value.
>   2. Modify the cleanup logic in ovs-agent not to delete flows
>  which have a reserved integer cookie value.

https://review.openstack.org/#/c/257285/
https://review.openstack.org/#/c/257286/

Thanks,
KAZ


From: Soichi Shigeta <shigeta.soi...@jp.fujitsu.com>
Subject: Re: [openstack-dev] [neutron][taas] neutron ovs-agent deletes taas 
flows
Date: Thu, 17 Dec 2015 09:28:10 +0900

> 
>  Thank you for your helpful comments.
> 
>  I'd like to update my proposal as follows:
> 
>   1. Set an integer cookie value to taas flows.
>  Maybe all "1" for short term tentative value.
>   2. Modify the cleanup logic in ovs-agent not to delete flows
>  which have a reserved integer cookie value.
> 
> 
>> Sorry, that I don't see this earlier. Yes, cookies have integer
>> values, so
>> we won't be able to set string there. May be we can have a reserved
>> integer
>> cookie value for a project like all "1".
>>
>> I won't support idea of modifying cleanup logic not to drop 0x0
>> cookies.
>> During implementation of graceful restart it was not dropped at first,
>> but
>> I get rid of it as having a lot of flows not related to anything was
>> not
>> desirable, so we should try to avoid it here, too.
>>
>> On Wed, Dec 16, 2015 at 7:46 AM, Soichi Shigeta <
>> shigeta.soi...@jp.fujitsu.com> wrote:
>>
>>>
>>> o) An idea to fix:
>>>>
>>>>1. Set "taas" stamp(*) to taas flows.
>>>>2. Modify the cleanup logic in ovs-agent not to delete entries
>>>>   stamped as "taas".
>>>>
>>>>* Maybe a static string.
>>>>  If we need to use a string which generated dynamically
>>>>  (e.g. uuid), API to interact with ovs-agent is required.
>>>>
>>>>
>>>Last week I proposed to set a static string (e.g. "taas") as cookie
>>>of flows created by taas agent.
>>>
>>>But I found that the value of a cookie should not be a string,
>>>but an integer.
>>>
>>>At line 187 in
>>> "neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py":
>>>self.agent_uuid_stamp = uuid.uuid4().int & UINT64_BITMASK
>>>
>>>In case of we set an integer value to cookie, coordination
>>>(reservation of range) is required to avoid conflict of cookies with
>>>other neutron sub-projects.
>>>
>>>As an alternative (*** short term ***) solution, my idea is:
>>>Modify the clean up logic in ovs agent not to delete flows whose
>>>"cookie = 0x0".
>>>Because old flows created by ovs agent have an old stamp, "cookie =
>>>0x0" means it was created by other than ovs agent.
>>>
>>># But, this idea has a disadvantage:
>>>  If there are flows which have been created by older version of ovs
>>>  agent, they can not be cleaned.
>>>
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][taas] neutron ovs-agent deletes taas flows

[Soichi Shigeta]

 Thank you for your helpful comments.

 I'd like to update my proposal as follows:

  1. Set an integer cookie value to taas flows.
 Maybe all "1" for short term tentative value.
  2. Modify the cleanup logic in ovs-agent not to delete flows
 which have a reserved integer cookie value.



Sorry, that I don't see this earlier. Yes, cookies have integer values, so
we won't be able to set string there. May be we can have a reserved integer
cookie value for a project like all "1".

I won't support idea of modifying cleanup logic not to drop 0x0 cookies.
During implementation of graceful restart it was not dropped at first, but
I get rid of it  as having a lot of flows not related to anything was not
desirable, so we should try to avoid it here, too.

On Wed, Dec 16, 2015 at 7:46 AM, Soichi Shigeta <
shigeta.soi...@jp.fujitsu.com> wrote:



o) An idea to fix:


   1. Set "taas" stamp(*) to taas flows.
   2. Modify the cleanup logic in ovs-agent not to delete entries
  stamped as "taas".

   * Maybe a static string.
 If we need to use a string which generated dynamically
 (e.g. uuid), API to interact with ovs-agent is required.



   Last week I proposed to set a static string (e.g. "taas") as cookie
   of flows created by taas agent.

   But I found that the value of a cookie should not be a string,
   but an integer.

   At line 187 in
"neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py":
   self.agent_uuid_stamp = uuid.uuid4().int & UINT64_BITMASK

   In case of we set an integer value to cookie, coordination
   (reservation of range) is required to avoid conflict of cookies with
   other neutron sub-projects.

   As an alternative (*** short term ***) solution, my idea is:
   Modify the clean up logic in ovs agent not to delete flows whose
   "cookie = 0x0".
   Because old flows created by ovs agent have an old stamp, "cookie =
   0x0" means it was created by other than ovs agent.

   # But, this idea has a disadvantage:
 If there are flows which have been created by older version of ovs
 agent, they can not be cleaned.





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][taas] neutron ovs-agent deletes taas flows

[Kevin Benton]

If you go this route, taas will have to implement flow recovery logic
itself as well if you want hitless restarts.

On Wed, Dec 16, 2015 at 5:28 PM, Soichi Shigeta <
shigeta.soi...@jp.fujitsu.com> wrote:

>
>  Thank you for your helpful comments.
>
>  I'd like to update my proposal as follows:
>
>   1. Set an integer cookie value to taas flows.
>  Maybe all "1" for short term tentative value.
>   2. Modify the cleanup logic in ovs-agent not to delete flows
>  which have a reserved integer cookie value.
>
>
>
> Sorry, that I don't see this earlier. Yes, cookies have integer values, so
>> we won't be able to set string there. May be we can have a reserved
>> integer
>> cookie value for a project like all "1".
>>
>> I won't support idea of modifying cleanup logic not to drop 0x0 cookies.
>> During implementation of graceful restart it was not dropped at first, but
>> I get rid of it  as having a lot of flows not related to anything was not
>> desirable, so we should try to avoid it here, too.
>>
>> On Wed, Dec 16, 2015 at 7:46 AM, Soichi Shigeta <
>> shigeta.soi...@jp.fujitsu.com> wrote:
>>
>>
>>> o) An idea to fix:
>>>

1. Set "taas" stamp(*) to taas flows.
2. Modify the cleanup logic in ovs-agent not to delete entries
   stamped as "taas".

* Maybe a static string.
  If we need to use a string which generated dynamically
  (e.g. uuid), API to interact with ovs-agent is required.


Last week I proposed to set a static string (e.g. "taas") as cookie
>>>of flows created by taas agent.
>>>
>>>But I found that the value of a cookie should not be a string,
>>>but an integer.
>>>
>>>At line 187 in
>>>
>>> "neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py":
>>>self.agent_uuid_stamp = uuid.uuid4().int & UINT64_BITMASK
>>>
>>>In case of we set an integer value to cookie, coordination
>>>(reservation of range) is required to avoid conflict of cookies with
>>>other neutron sub-projects.
>>>
>>>As an alternative (*** short term ***) solution, my idea is:
>>>Modify the clean up logic in ovs agent not to delete flows whose
>>>"cookie = 0x0".
>>>Because old flows created by ovs agent have an old stamp, "cookie =
>>>0x0" means it was created by other than ovs agent.
>>>
>>># But, this idea has a disadvantage:
>>>  If there are flows which have been created by older version of ovs
>>>  agent, they can not be cleaned.
>>>
>>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Kevin Benton
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][taas] neutron ovs-agent deletes taas flows

[Ihar Hrachyshka]

Kyle Mestery  wrote:


>>   o) Workaround:
>>
>>  After a vm is deployed on a (re)started compute node, restart taas
>>  agent before creating a tap-service or tap-flow.
>>  That is, create taas flows after cleanup has been done.
>>
>>  Note that cleanup will be done only once during an ovs-agent is
>>  running.
>>
>>
>>   o) An idea to fix:
>>
>>  1. Set "taas" stamp(*) to taas flows.
>>  2. Modify the cleanup logic in ovs-agent not to delete entries
>> stamped as "taas".
>>
>>  * Maybe a static string.
>>If we need to use a string which generated dynamically
>>(e.g. uuid), API to interact with ovs-agent is required.
>
>
> API proposal with some consideration for flow cleanup not dropping  
flows for

> external code is covered in the following email thread:
>  
http://lists.openstack.org/pipermail/openstack-dev/2015-December/081264.html

>
> I believe you would need to adopt the extensions API once it’s in, moving
> from setup with a separate agent for your feature to l2 agent extension  
for

> taas that will run inside OVS agent.
>

This is really the right approach here as well. Anything modifying flow  
tables and expecting to work with the OVS L2 agent should in fact reside  
in the OVS L2 agent or use this extension API. Ihar, is this currently  
planned for Mitaka?


Optimistically, yes. But I currently lack comments on the approach before I  
bake a spec for that. I would really like to hear from OVS agent folks  
(Rossella, Ann, who else?) on whether the proposal is sane at least on high  
level. I know some folks were promising quick replies to the thread.


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][taas] neutron ovs-agent deletes taas flows

[Ihar Hrachyshka]

Assaf Muller  wrote:


SFC are going to hit this issue as well. Really any out of tree
Neutron project that extends the OVS agent and expects things to work
:)


Yes. SFC project is considered for l2 agent extensions mechanism, though we  
need to deliver in core first. AFAIK for the time being they ship their OVS  
agent fork (or plan to). Other projects that we are aware of being affected  
are BGP-VPN, or some new QoS rule types.


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][taas] neutron ovs-agent deletes taas flows

[Soichi Shigeta]

   o) An idea to fix:

  1. Set "taas" stamp(*) to taas flows.
  2. Modify the cleanup logic in ovs-agent not to delete entries
 stamped as "taas".

  * Maybe a static string.
If we need to use a string which generated dynamically
(e.g. uuid), API to interact with ovs-agent is required.



  Last week I proposed to set a static string (e.g. "taas") as cookie
  of flows created by taas agent.

  But I found that the value of a cookie should not be a string,
  but an integer.

  At line 187 in 
"neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py":

  self.agent_uuid_stamp = uuid.uuid4().int & UINT64_BITMASK

  In case of we set an integer value to cookie, coordination
  (reservation of range) is required to avoid conflict of cookies with
  other neutron sub-projects.

  As an alternative (*** short term ***) solution, my idea is:
  Modify the clean up logic in ovs agent not to delete flows whose
  "cookie = 0x0".
  Because old flows created by ovs agent have an old stamp, "cookie =
  0x0" means it was created by other than ovs agent.

  # But, this idea has a disadvantage:
If there are flows which have been created by older version of ovs
agent, they can not be cleaned.

---
 Soichi Shigeta




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][taas] neutron ovs-agent deletes taas flows

[Ihar Hrachyshka]

Soichi Shigeta  wrote:



 Hi,

  We find a problem that neutron ovs-agent deletes taas flows.

  o) Problem description:

 Background:
  At Liberty, a bug fix to drop only old flows was merged
  to Neutron.
  When ovs-agent is restarted, the cleanup logic drops flow
  entries unless they are stamped by agent_uuid (recorded as
  a cookie).

  bug: #1383674
   "Restarting neutron openvswitch agent causes network
hiccup by throwing away all flows"
   https://bugs.launchpad.net/neutron/+bug/1383674

  commit: 73673beacd75a2d9f51f15b284f1b458d32e992e (patch)
https://git.openstack.org/cgit/openstack/neutron/commit/?id=73673beacd75a2d9f51f15b284f1b458d32e992e


 Problem:
  Cleanup will be done only once, but it seems not to work
  until port configuration is changed.

  Therefore, taas flows will be deleted as follows:
   1. Start a new compute node or restart an existing compute node.
   2. Start taas agent on the compute node.
  --> taas agent creates flows
  (these flows are not stamped by using ovs-agent's uuid)
   3. Deploy a vm on the compute node.
  --> 1. neutron changes port configuration
  2. subsequently, the cleanup logic is invoked
  3. ovs-agent drops taas flows

 Specifically, following taas flows in br_tun are dropped:
 -
  table=35, priority=2,reg0=0x0 actions=resubmit(,36)
  table=35, priority=1,reg0=0x1 actions=resubmit(,36)
  table=35, priority=1,reg0=0x2 actions=resubmit(,37)
 -

 log in q-agt.log
 -
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
cookie=0x0, duration=434.59s, table=35, n_packets=0, n_bytes=0,
idle_age=434, priority=2,reg0=0x0 actions=resubmit(,36)
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
cookie=0x0, duration=434.587s, table=35, n_packets=0, n_bytes=0,
idle_age=434, priority=1,reg0=0x1 actions=resubmit(,36)
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
cookie=0x0, duration=434.583s, table=35, n_packets=0, n_bytes=0,
idle_age=434, priority=1,reg0=0x2 actions=resubmit(,37)
 -


  o) Impact for TaaS:

 Because flows in br_tun are dropped by the cleanup logic, mirrored
 packets will not send to a monitoring vm running on another host.

 Note: Mirrored packets are sent in case of both source vm and
   monitoring vm are running on the same host. (not affected by
   flows in br_tun)


  o) How to reproduce:

 1. Start a new compute node or restart an existing compute node.
(Actually, restarting ovs-agent is enough.)
 2. Start (or restart) taas agent on the compute node.
 3. Deploy a vm on the compute node.
--> The cleanup logic drops taas flows.


  o) Workaround:

 After a vm is deployed on a (re)started compute node, restart taas
 agent before creating a tap-service or tap-flow.
 That is, create taas flows after cleanup has been done.

 Note that cleanup will be done only once during an ovs-agent is
 running.


  o) An idea to fix:

 1. Set "taas" stamp(*) to taas flows.
 2. Modify the cleanup logic in ovs-agent not to delete entries
stamped as "taas".

 * Maybe a static string.
   If we need to use a string which generated dynamically
   (e.g. uuid), API to interact with ovs-agent is required.


API proposal with some consideration for flow cleanup not dropping flows  
for external code is covered in the following email thread:  
http://lists.openstack.org/pipermail/openstack-dev/2015-December/081264.html


I believe you would need to adopt the extensions API once it’s in, moving  
from setup with a separate agent for your feature to l2 agent extension for  
taas that will run inside OVS agent.


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][taas] neutron ovs-agent deletes taas flows

[Assaf Muller]

SFC are going to hit this issue as well. Really any out of tree
Neutron project that extends the OVS agent and expects things to work
:)

On Tue, Dec 15, 2015 at 9:30 AM, Ihar Hrachyshka  wrote:
> Soichi Shigeta  wrote:
>
>>
>>  Hi,
>>
>>   We find a problem that neutron ovs-agent deletes taas flows.
>>
>>   o) Problem description:
>>
>>  Background:
>>   At Liberty, a bug fix to drop only old flows was merged
>>   to Neutron.
>>   When ovs-agent is restarted, the cleanup logic drops flow
>>   entries unless they are stamped by agent_uuid (recorded as
>>   a cookie).
>>
>>   bug: #1383674
>>"Restarting neutron openvswitch agent causes network
>> hiccup by throwing away all flows"
>>https://bugs.launchpad.net/neutron/+bug/1383674
>>
>>   commit: 73673beacd75a2d9f51f15b284f1b458d32e992e (patch)
>>
>> https://git.openstack.org/cgit/openstack/neutron/commit/?id=73673beacd75a2d9f51f15b284f1b458d32e992e
>>
>>
>>  Problem:
>>   Cleanup will be done only once, but it seems not to work
>>   until port configuration is changed.
>>
>>   Therefore, taas flows will be deleted as follows:
>>1. Start a new compute node or restart an existing compute node.
>>2. Start taas agent on the compute node.
>>   --> taas agent creates flows
>>   (these flows are not stamped by using ovs-agent's uuid)
>>3. Deploy a vm on the compute node.
>>   --> 1. neutron changes port configuration
>>   2. subsequently, the cleanup logic is invoked
>>   3. ovs-agent drops taas flows
>>
>>  Specifically, following taas flows in br_tun are dropped:
>>  -
>>   table=35, priority=2,reg0=0x0 actions=resubmit(,36)
>>   table=35, priority=1,reg0=0x1 actions=resubmit(,36)
>>   table=35, priority=1,reg0=0x2 actions=resubmit(,37)
>>  -
>>
>>  log in q-agt.log
>>  -
>> neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
>> req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
>> cookie=0x0, duration=434.59s, table=35, n_packets=0, n_bytes=0,
>> idle_age=434, priority=2,reg0=0x0 actions=resubmit(,36)
>> neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
>> req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
>> cookie=0x0, duration=434.587s, table=35, n_packets=0, n_bytes=0,
>> idle_age=434, priority=1,reg0=0x1 actions=resubmit(,36)
>> neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
>> req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
>> cookie=0x0, duration=434.583s, table=35, n_packets=0, n_bytes=0,
>> idle_age=434, priority=1,reg0=0x2 actions=resubmit(,37)
>>  -
>>
>>
>>   o) Impact for TaaS:
>>
>>  Because flows in br_tun are dropped by the cleanup logic, mirrored
>>  packets will not send to a monitoring vm running on another host.
>>
>>  Note: Mirrored packets are sent in case of both source vm and
>>monitoring vm are running on the same host. (not affected by
>>flows in br_tun)
>>
>>
>>   o) How to reproduce:
>>
>>  1. Start a new compute node or restart an existing compute node.
>> (Actually, restarting ovs-agent is enough.)
>>  2. Start (or restart) taas agent on the compute node.
>>  3. Deploy a vm on the compute node.
>> --> The cleanup logic drops taas flows.
>>
>>
>>   o) Workaround:
>>
>>  After a vm is deployed on a (re)started compute node, restart taas
>>  agent before creating a tap-service or tap-flow.
>>  That is, create taas flows after cleanup has been done.
>>
>>  Note that cleanup will be done only once during an ovs-agent is
>>  running.
>>
>>
>>   o) An idea to fix:
>>
>>  1. Set "taas" stamp(*) to taas flows.
>>  2. Modify the cleanup logic in ovs-agent not to delete entries
>> stamped as "taas".
>>
>>  * Maybe a static string.
>>If we need to use a string which generated dynamically
>>(e.g. uuid), API to interact with ovs-agent is required.
>
>
> API proposal with some consideration for flow cleanup not dropping flows for
> external code is covered in the following email thread:
> http://lists.openstack.org/pipermail/openstack-dev/2015-December/081264.html
>
> I believe you would need to adopt the extensions API once it’s in, moving
> from setup with a separate agent for your feature to l2 agent extension for
> taas that will run inside OVS agent.
>
> Ihar
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][taas] neutron ovs-agent deletes taas flows

[Kyle Mestery]

On Tue, Dec 15, 2015 at 9:11 AM, Assaf Muller  wrote:

> SFC are going to hit this issue as well. Really any out of tree
> Neutron project that extends the OVS agent and expects things to work
> :)
>
>
Yes, this is the case.


> On Tue, Dec 15, 2015 at 9:30 AM, Ihar Hrachyshka 
> wrote:
> > Soichi Shigeta  wrote:
> >
> >>
> >>  Hi,
> >>
> >>   We find a problem that neutron ovs-agent deletes taas flows.
> >>
> >>   o) Problem description:
> >>
> >>  Background:
> >>   At Liberty, a bug fix to drop only old flows was merged
> >>   to Neutron.
> >>   When ovs-agent is restarted, the cleanup logic drops flow
> >>   entries unless they are stamped by agent_uuid (recorded as
> >>   a cookie).
> >>
> >>   bug: #1383674
> >>"Restarting neutron openvswitch agent causes network
> >> hiccup by throwing away all flows"
> >>https://bugs.launchpad.net/neutron/+bug/1383674
> >>
> >>   commit: 73673beacd75a2d9f51f15b284f1b458d32e992e (patch)
> >>
> >>
> https://git.openstack.org/cgit/openstack/neutron/commit/?id=73673beacd75a2d9f51f15b284f1b458d32e992e
> >>
> >>
> >>  Problem:
> >>   Cleanup will be done only once, but it seems not to work
> >>   until port configuration is changed.
> >>
> >>   Therefore, taas flows will be deleted as follows:
> >>1. Start a new compute node or restart an existing compute node.
> >>2. Start taas agent on the compute node.
> >>   --> taas agent creates flows
> >>   (these flows are not stamped by using ovs-agent's uuid)
> >>3. Deploy a vm on the compute node.
> >>   --> 1. neutron changes port configuration
> >>   2. subsequently, the cleanup logic is invoked
> >>   3. ovs-agent drops taas flows
> >>
> >>  Specifically, following taas flows in br_tun are dropped:
> >>  -
> >>   table=35, priority=2,reg0=0x0 actions=resubmit(,36)
> >>   table=35, priority=1,reg0=0x1 actions=resubmit(,36)
> >>   table=35, priority=1,reg0=0x2 actions=resubmit(,37)
> >>  -
> >>
> >>  log in q-agt.log
> >>  -
> >>
> neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
> >> req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
> >> cookie=0x0, duration=434.59s, table=35, n_packets=0, n_bytes=0,
> >> idle_age=434, priority=2,reg0=0x0 actions=resubmit(,36)
> >>
> neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
> >> req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
> >> cookie=0x0, duration=434.587s, table=35, n_packets=0, n_bytes=0,
> >> idle_age=434, priority=1,reg0=0x1 actions=resubmit(,36)
> >>
> neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
> >> req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
> >> cookie=0x0, duration=434.583s, table=35, n_packets=0, n_bytes=0,
> >> idle_age=434, priority=1,reg0=0x2 actions=resubmit(,37)
> >>  -
> >>
> >>
> >>   o) Impact for TaaS:
> >>
> >>  Because flows in br_tun are dropped by the cleanup logic, mirrored
> >>  packets will not send to a monitoring vm running on another host.
> >>
> >>  Note: Mirrored packets are sent in case of both source vm and
> >>monitoring vm are running on the same host. (not affected by
> >>flows in br_tun)
> >>
> >>
> >>   o) How to reproduce:
> >>
> >>  1. Start a new compute node or restart an existing compute node.
> >> (Actually, restarting ovs-agent is enough.)
> >>  2. Start (or restart) taas agent on the compute node.
> >>  3. Deploy a vm on the compute node.
> >> --> The cleanup logic drops taas flows.
> >>
> >>
> >>   o) Workaround:
> >>
> >>  After a vm is deployed on a (re)started compute node, restart taas
> >>  agent before creating a tap-service or tap-flow.
> >>  That is, create taas flows after cleanup has been done.
> >>
> >>  Note that cleanup will be done only once during an ovs-agent is
> >>  running.
> >>
> >>
> >>   o) An idea to fix:
> >>
> >>  1. Set "taas" stamp(*) to taas flows.
> >>  2. Modify the cleanup logic in ovs-agent not to delete entries
> >> stamped as "taas".
> >>
> >>  * Maybe a static string.
> >>If we need to use a string which generated dynamically
> >>(e.g. uuid), API to interact with ovs-agent is required.
> >
> >
> > API proposal with some consideration for flow cleanup not dropping flows
> for
> > external code is covered in the following email thread:
> >
> http://lists.openstack.org/pipermail/openstack-dev/2015-December/081264.html
> >
> > I believe you would need to adopt the extensions API once it’s in, moving
> > from setup with a separate agent for your feature to l2 agent extension
> for
> > taas that will run inside OVS agent.
> >
>

This is 

Re: [openstack-dev] [neutron][taas] neutron ovs-agent deletes taas flows

[Anna Kamyshnikova]

Sorry, that I don't see this earlier. Yes, cookies have integer values, so
we won't be able to set string there. May be we can have a reserved integer
cookie value for a project like all "1".

I won't support idea of modifying cleanup logic not to drop 0x0 cookies.
During implementation of graceful restart it was not dropped at first, but
I get rid of it  as having a lot of flows not related to anything was not
desirable, so we should try to avoid it here, too.

On Wed, Dec 16, 2015 at 7:46 AM, Soichi Shigeta <
shigeta.soi...@jp.fujitsu.com> wrote:

>
>o) An idea to fix:
>>
>>   1. Set "taas" stamp(*) to taas flows.
>>   2. Modify the cleanup logic in ovs-agent not to delete entries
>>  stamped as "taas".
>>
>>   * Maybe a static string.
>> If we need to use a string which generated dynamically
>> (e.g. uuid), API to interact with ovs-agent is required.
>>
>>
>   Last week I proposed to set a static string (e.g. "taas") as cookie
>   of flows created by taas agent.
>
>   But I found that the value of a cookie should not be a string,
>   but an integer.
>
>   At line 187 in
> "neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py":
>   self.agent_uuid_stamp = uuid.uuid4().int & UINT64_BITMASK
>
>   In case of we set an integer value to cookie, coordination
>   (reservation of range) is required to avoid conflict of cookies with
>   other neutron sub-projects.
>
>   As an alternative (*** short term ***) solution, my idea is:
>   Modify the clean up logic in ovs agent not to delete flows whose
>   "cookie = 0x0".
>   Because old flows created by ovs agent have an old stamp, "cookie =
>   0x0" means it was created by other than ovs agent.
>
>   # But, this idea has a disadvantage:
> If there are flows which have been created by older version of ovs
> agent, they can not be cleaned.
>
>
> ---
>  Soichi Shigeta
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Regards,
Ann Kamyshnikova
Mirantis, Inc
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron][taas] neutron ovs-agent deletes taas flows

[Soichi Shigeta]

 Hi,

  We find a problem that neutron ovs-agent deletes taas flows.

  o) Problem description:

 Background:
  At Liberty, a bug fix to drop only old flows was merged
  to Neutron.
  When ovs-agent is restarted, the cleanup logic drops flow
  entries unless they are stamped by agent_uuid (recorded as
  a cookie).

  bug: #1383674
   "Restarting neutron openvswitch agent causes network
hiccup by throwing away all flows"
   https://bugs.launchpad.net/neutron/+bug/1383674

  commit: 73673beacd75a2d9f51f15b284f1b458d32e992e (patch)

https://git.openstack.org/cgit/openstack/neutron/commit/?id=73673beacd75a2d9f51f15b284f1b458d32e992e


 Problem:
  Cleanup will be done only once, but it seems not to work
  until port configuration is changed.

  Therefore, taas flows will be deleted as follows:
   1. Start a new compute node or restart an existing compute node.
   2. Start taas agent on the compute node.
  --> taas agent creates flows
  (these flows are not stamped by using ovs-agent's uuid)
   3. Deploy a vm on the compute node.
  --> 1. neutron changes port configuration
  2. subsequently, the cleanup logic is invoked
  3. ovs-agent drops taas flows

 Specifically, following taas flows in br_tun are dropped:
 -
  table=35, priority=2,reg0=0x0 actions=resubmit(,36)
  table=35, priority=1,reg0=0x1 actions=resubmit(,36)
  table=35, priority=1,reg0=0x2 actions=resubmit(,37)
 -

 log in q-agt.log
 -

neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
cookie=0x0, duration=434.59s, table=35, n_packets=0, n_bytes=0,
idle_age=434, priority=2,reg0=0x0 actions=resubmit(,36)

neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
cookie=0x0, duration=434.587s, table=35, n_packets=0, n_bytes=0,
idle_age=434, priority=1,reg0=0x1 actions=resubmit(,36)

neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
cookie=0x0, duration=434.583s, table=35, n_packets=0, n_bytes=0,
idle_age=434, priority=1,reg0=0x2 actions=resubmit(,37)
 -


  o) Impact for TaaS:

 Because flows in br_tun are dropped by the cleanup logic, mirrored
 packets will not send to a monitoring vm running on another host.

 Note: Mirrored packets are sent in case of both source vm and
   monitoring vm are running on the same host. (not affected by
   flows in br_tun)


  o) How to reproduce:

 1. Start a new compute node or restart an existing compute node.
(Actually, restarting ovs-agent is enough.)
 2. Start (or restart) taas agent on the compute node.
 3. Deploy a vm on the compute node.
--> The cleanup logic drops taas flows.


  o) Workaround:

 After a vm is deployed on a (re)started compute node, restart taas
 agent before creating a tap-service or tap-flow.
 That is, create taas flows after cleanup has been done.

 Note that cleanup will be done only once during an ovs-agent is
 running.


  o) An idea to fix:

 1. Set "taas" stamp(*) to taas flows.
 2. Modify the cleanup logic in ovs-agent not to delete entries
stamped as "taas".

 * Maybe a static string.
   If we need to use a string which generated dynamically
   (e.g. uuid), API to interact with ovs-agent is required.


 ---
 Soichi Shigeta



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][taas] neutron ovs-agent deletes taas flows

[Takashi Yamamoto]

hi,

On Tue, Dec 8, 2015 at 6:52 PM, Soichi Shigeta
 wrote:
>
>  Hi,
>
>   We find a problem that neutron ovs-agent deletes taas flows.
>
>   o) Problem description:
>
>  Background:
>   At Liberty, a bug fix to drop only old flows was merged
>   to Neutron.
>   When ovs-agent is restarted, the cleanup logic drops flow
>   entries unless they are stamped by agent_uuid (recorded as
>   a cookie).
>
>   bug: #1383674
>"Restarting neutron openvswitch agent causes network
> hiccup by throwing away all flows"
>https://bugs.launchpad.net/neutron/+bug/1383674
>
>   commit: 73673beacd75a2d9f51f15b284f1b458d32e992e (patch)
>
> https://git.openstack.org/cgit/openstack/neutron/commit/?id=73673beacd75a2d9f51f15b284f1b458d32e992e
>
>
>  Problem:
>   Cleanup will be done only once, but it seems not to work
>   until port configuration is changed.
>
>   Therefore, taas flows will be deleted as follows:
>1. Start a new compute node or restart an existing compute node.
>2. Start taas agent on the compute node.
>   --> taas agent creates flows
>   (these flows are not stamped by using ovs-agent's uuid)
>3. Deploy a vm on the compute node.
>   --> 1. neutron changes port configuration
>   2. subsequently, the cleanup logic is invoked
>   3. ovs-agent drops taas flows
>
>  Specifically, following taas flows in br_tun are dropped:
>  -
>   table=35, priority=2,reg0=0x0 actions=resubmit(,36)
>   table=35, priority=1,reg0=0x1 actions=resubmit(,36)
>   table=35, priority=1,reg0=0x2 actions=resubmit(,37)
>  -
>
>  log in q-agt.log
>  -
>
> neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
> req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
> cookie=0x0, duration=434.59s, table=35, n_packets=0, n_bytes=0,
> idle_age=434, priority=2,reg0=0x0 actions=resubmit(,36)
>
> neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
> req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
> cookie=0x0, duration=434.587s, table=35, n_packets=0, n_bytes=0,
> idle_age=434, priority=1,reg0=0x1 actions=resubmit(,36)
>
> neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch
> req-e5739280-7116-4802-b5ba-d6964b4c5557 Deleting flow
> cookie=0x0, duration=434.583s, table=35, n_packets=0, n_bytes=0,
> idle_age=434, priority=1,reg0=0x2 actions=resubmit(,37)
>  -
>
>
>   o) Impact for TaaS:
>
>  Because flows in br_tun are dropped by the cleanup logic, mirrored
>  packets will not send to a monitoring vm running on another host.
>
>  Note: Mirrored packets are sent in case of both source vm and
>monitoring vm are running on the same host. (not affected by
>flows in br_tun)
>
>
>   o) How to reproduce:
>
>  1. Start a new compute node or restart an existing compute node.
> (Actually, restarting ovs-agent is enough.)
>  2. Start (or restart) taas agent on the compute node.
>  3. Deploy a vm on the compute node.
> --> The cleanup logic drops taas flows.

i've heard qos folks had to deal with the same issues.

>
>
>   o) Workaround:
>
>  After a vm is deployed on a (re)started compute node, restart taas
>  agent before creating a tap-service or tap-flow.
>  That is, create taas flows after cleanup has been done.
>
>  Note that cleanup will be done only once during an ovs-agent is
>  running.
>
>
>   o) An idea to fix:
>
>  1. Set "taas" stamp(*) to taas flows.
>  2. Modify the cleanup logic in ovs-agent not to delete entries
> stamped as "taas".
>
>  * Maybe a static string.
>If we need to use a string which generated dynamically
>(e.g. uuid), API to interact with ovs-agent is required.
>
>
>  ---
>  Soichi Shigeta
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev