[openstack-dev] [neutron] heads up to long time ovs users...

[IWAMOTO Toshihiro]

The neutron team is finally removing the ovs-ofctl option.

https://review.openstack.org/#/c/599496/

The ovs-ofctl of_interface option wasn't default since Newton and was
deprecated in Pike.

So, if you are a long time ovs-agent user and upgrading to a new
coming release, you must switch from the ovs-ofctl implementation to
the native implementation and are affected by the following issue.

https://bugs.launchpad.net/neutron/+bug/1793354

The loss of communication mentioned in this bug report would be a few
seconds to a few minutes depending on the number of network
interfaces.  It happens when an ovs-agent is restarted with the new
of_interface (so only once during the upgrade) and persists until the
network interfaces are set up.

Please speak up if you cannot tolerate this during upgrades.

IIUC, this bug is unfixable and I'd like to move forward as
maintaining two of_interface implementation is a burden for the
neutron team.

--
IWAMOTO Toshihiro


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [python3] flake8 and pycodestyle W60x warnings

[IWAMOTO Toshihiro]

pycodestyle-2.4.0 added new warnings W605 and W606, which needs to be
addressed or future versions of python3 will refuse to run.

https://github.com/PyCQA/pycodestyle/pull/676

(OpenStack CI's flake8 is pre pycodestyle age, flake8 version isn't
managed by g-r, but that's another story. No release of flake8
supports pycodestyle 2.4.0 yet. ;)

nova seems to have ~200 of those warnings, while other projects don't
have much, FWIW.

--
IWAMOTO Toshihiro

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][dynamic routing] RYU Breaks lower constraints

[IWAMOTO Toshihiro]

On Sun, 15 Apr 2018 21:02:42 +0900,
Gary Kotton wrote:
> 
> [1  ]
> [1.1  ]
> Hi,
> That sounds reasonable. I wonder if the RYU folk can chime in here.
> Thanks

I don't fully understand the recent g-r change yet, but
I guess neutron-dynamic-routing should  also have ryu>=4.24.
I'll check this tommorrow.

> From: Akihiro MOTOKI 
> Reply-To: OpenStack List 
> Date: Sunday, April 15, 2018 at 12:43 PM
> To: OpenStack List 
> Subject: Re: [openstack-dev] [neutron][dynamic routing] RYU Breaks lower 
> constraints
> 
> Gary,
> 
> I think this is caused by the recent pip change and pip no longer cannot 
> import pip from code. The right solution seems to bump the minimum version of 
> ryu.
> 
> Thought?
> 
> http://lists.openstack.org/pipermail/openstack-dev/2018-March/128939.html
> 
> Akihiro
> 
> 2018/04/15 午後6:06 "Gary Kotton" 
> >:
> Hi,
> It seems like ther RYU import is breaking the project:
> 
> 
> 2018-04-15 
> 08:41:34.654681
>  | ubuntu-xenial | b'--- import errors ---\nFailed to import test module: 
> neutron_dynamic_routing.tests.unit.services.bgp.driver.ryu.test_driver\nTraceback
>  (most recent call last):\n  File 
> "/home/zuul/src/git.openstack.org/openstack/neutron-dynamic-routing/.tox/lower-constraints/lib/python3.5/site-packages/unittest2/loader.py",
>  line 456, in _find_test_path\nmodule = 
> self._get_module_from_name(name)\n  File 
> "/home/zuul/src/git.openstack.org/openstack/neutron-dynamic-routing/.tox/lower-constraints/lib/python3.5/site-packages/unittest2/loader.py",
>  line 395, in _get_modu
 le_from_name\n__import__(name)\n  File 
"/home/zuul/src/git.openstack.org/openstack/neutron-dynamic-routing/neutron_dynamic_routing/tests/unit/services/bgp/driver/ryu/test_driver.py",
 line 21, in \nfrom ryu.services.protocols.bgp import bgpspeaker\n  
File 
"/home/zuul/src/git.openstack.org/openstack/neutron-dynamic-routing/.tox/lower-constraints/lib/python3.5/site-packages/ryu/services/protocols/bgp/bgpspeaker.py",
 line 21, in \nfrom ryu.lib.packet.bgp import (\n  File 
"/home/zuul/src/git.openstack.org/openstack/neutron-dynamic-routing/.tox/lower-constraints/lib/python3.5/site-packages/ryu/lib/packet/__init__.py", 
line 6, in \nfrom . import (ethernet, arp, icmp, icmpv6, ipv4, 
ipv6, lldp, mpls, packet,\n  File 
"/home/zuul/src/git.openstack.org/openstack/neutron-dynamic-routing/.tox/lower-constraints/lib/python3.5/site-packages/ryu/lib/packet/ethernet.py",
 line 18, in \nfrom . import vlan\n  File 
"/home/zuul/src/git.openstack.org/openstack/neutron-dynamic-routing/.tox/lower-constraints/lib/python3.5/site-packages/ryu/lib/packet/vlan.py",
 line 21, in \nfrom . import ipv4\n  File 
"/home/zuul/src/git.openstack.org/openstack/neutron-dynamic-routing/.tox/lower-constraints/lib/python3.5/site-packages/ryu/lib/packet/ipv4.py",
 line 23, in \nfrom . import tcp\n  File 
"/home/zuul/src/git.openstack.org/openstack/neutron-dynamic-routing/.tox/lower-constraints/lib/python3.5/site-packages/ryu/lib/packet/tcp.py",
 line 24, in \nfrom . import bgp\n  File 
"/home/zuul/src/git.openstack.org/openstack/neutron-dynamic-routing/.tox/lower-constraints/lib/python3.5/site-packages/ryu/lib/packet/bgp.py",
 line 52, in \nfrom ryu.utils import binary_str\n  File 
"/home/zuul/src/git.openstack.org/openstack/neutron-dynamic-routing/.tox/lower-constraints/lib/python3.5/site-packages/ryu/utils.py",
 line 23, in \nfrom pip import req as pip_req\nImportError: cannot 
import name \'req\'\n'
> 
> Any suggestions?

Re: [openstack-dev] [all][requirements] uncapping eventlet

[IWAMOTO Toshihiro]

On Mon, 09 Apr 2018 22:58:28 +0900,
Doug Hellmann wrote:
> 
> Excerpts from Tony Breeds's message of 2018-04-09 13:39:30 +1000:
> > On Fri, Apr 06, 2018 at 09:41:07AM -0700, Clark Boylan wrote:
> > 
> > > My understanding of our use of upper constraints was that this should
> > > (almost) always be the case for (almost) all dependencies.  We should
> > > rely on constraints instead of requirements caps. Capping libs like
> > > pbr or eventlet and any other that is in use globally is incredibly
> > > difficult to work with when you want to uncap it because you have to
> > > coordinate globally. Instead if using constraints you just bump the
> > > constraint and are done.
> > 
> > Part of the reason that we have the caps it to prevent the tools that
> > auto-generate the constraints syncs from considering these versions and
> > then depending on the requirements team to strip that from the bot
> > change before committing (assuming it passes CI).
> > 
> > Once the work Doug's doing is complete we could consider tweaking the
> > tools to use a different mechanism, but that's only part of the reason
> > for the caps in g-r.
> > 
> > Yours Tony.
> 
> Now that projects don't have to match the global requirements list
> entries exactly we should be able to remove caps from within the
> projects and keep caps in the global list for cases like this where we
> know we frequently encounter breaking changes in new releases. The
> changes to support that were part of
> https://review.openstack.org/#/c/555402/

As eventlet has been uncapped in g-r, requirements-check is
complaining on unrelated project-local requirement changes.
I'm not quite sure but doesn't seem to be a intended behavior.

http://logs.openstack.org/57/451257/16/check/requirements-check/c32ee69/job-output.txt.gz

--
IWAMOTO Toshihiro

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Denver Team Dinner

[IWAMOTO Toshihiro]

+1
thanks for organizing!

On Wed, 13 Sep 2017 14:18:45 +0900,
Brian Haley wrote:
> 
> +1
> 
> On 09/12/2017 10:44 PM, Ihar Hrachyshka wrote:
> > +1
> > 
> > On Tue, Sep 12, 2017 at 9:44 PM, Kevin Benton  wrote:
> >> +1
> >> 
> >> On Tue, Sep 12, 2017 at 8:50 PM, Sławek Kapłoński 
> >> wrote:
> >>> 
> >>> +1
> >>> 
> >>> —
> >>> Best regards
> >>> Slawek Kaplonski
> >>> sla...@kaplonski.pl
> >>> 
> >>> 
> >>> 
> >>> 
>  Wiadomość napisana przez Miguel Lavalle  w dniu
>  12.09.2017, o godz. 17:23:
>  
>  Dear Neutrinos,
>  
>  Our social event will take place on Thursday September 12th at 7:30pm.
>  The venue is going to be https://www.famousdaves.com/Denver-Stapleton. 
>  It is
>  located 0.4 miles from the Renaissance Hotel, which translates to an 
>  easy 9
>  minutes walk.
>  
>  I have a reservation for a group of 30 people under my name. Please
>  respond to this message with your attendance confirmation by Wednesday
>  night, so I can get a more accurate head count.
>  
>  Looking forward to see y'all Thursday night
>  
>  Best regards
>  
>  Miguel
>  
>  __
>  OpenStack Development Mailing List (not for usage questions)
>  Unsubscribe:
>  openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>> 
> >>> 
> >>> __
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>> 
> >> 
> >> 
> >> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> 
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] [gate] [all] openstack services footprint lead to oom-kill in the gate

[IWAMOTO Toshihiro]

At Wed, 1 Feb 2017 16:24:54 -0800,
Armando M. wrote:
> 
> Hi,
> 
> [TL;DR]: OpenStack services have steadily increased their memory
> footprints. We need a concerted way to address the oom-kills experienced in
> the openstack gate, as we may have reached a ceiling.
> 
> Now the longer version:
> 
> 
> We have been experiencing some instability in the gate lately due to a
> number of reasons. When everything adds up, this means it's rather
> difficult to merge anything and knowing we're in feature freeze, that adds
> to stress. One culprit was identified to be [1].
> 
> We initially tried to increase the swappiness, but that didn't seem to
> help. Then we have looked at the resident memory in use. When going back
> over the past three releases we have noticed that the aggregated memory
> footprint of some openstack projects has grown steadily. We have the
> following:

Not sure if it is due to memory shortage, VMs running CI jobs are
experiencing sluggishness, which may be the cause of ovs related
timeouts[1]. Tempest jobs run dstat to collect system info every
second. When timeouts[1] happen, dstat outputs are also often missing
for several seconds, which means a VM is having trouble scheduling
both ovs related processes and the dstat process.
Those ovs timeouts affect every project and happen much often than the
oom-kills.

Some details are on the lp bug page[2].

Correlation of such sluggishness and VM paging activities are not
clear. I wonder if VM hosts are under high load or if increasing VM
memory would help. Those VMs have no free ram for file cache and file
pages are read again and again, leading to extra IO loads on VM hosts
and adversely affecting other VMs on the same host.


[1] 
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22no%20response%20to%20inactivity%20probe%5C%22
[2] https://bugs.launchpad.net/neutron/+bug/1627106/comments/14

--
IWAMOTO Toshihiro

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] [gate] [all] openstack services footprint lead to oom-kill in the gate

[IWAMOTO Toshihiro]

At Wed, 1 Feb 2017 17:37:34 -0700,
Kevin Benton wrote:
> 
> [1  ]
> [1.1  ]
> And who said openstack wasn't growing? ;)
> 
> I think reducing API workers is a nice quick way to bring back some
> stability.
> 
> I have spent a bunch of time digging into the OOM killer events and haven't
> yet figured out why they are being triggered. There is significant swap
> space remaining in all of the cases I have seen so it's likely some memory

We can try increasing watermark_scale_factor instead.
I looked at random 2 oom-killer invocations but free mem were above
watermark. oom-killer were triggered by 16kB contig page alloation by
apparmor_file_alloc_security, so if we can try disabling apparmor that
may also work.


> locking issue or kernel allocations blocking swap. Until we can figure out
> the cause, we effectively have no usable swap space on the test instances
> so we are limited to 8GB.
> 
> On Feb 1, 2017 17:27, "Armando M."  wrote:
> 
> > Hi,
> >
> > [TL;DR]: OpenStack services have steadily increased their memory
> > footprints. We need a concerted way to address the oom-kills experienced in
> > the openstack gate, as we may have reached a ceiling.
> >
> > Now the longer version:
> > 
> >
> > We have been experiencing some instability in the gate lately due to a
> > number of reasons. When everything adds up, this means it's rather
> > difficult to merge anything and knowing we're in feature freeze, that adds
> > to stress. One culprit was identified to be [1].
> >
> > We initially tried to increase the swappiness, but that didn't seem to
> > help. Then we have looked at the resident memory in use. When going back
> > over the past three releases we have noticed that the aggregated memory
> > footprint of some openstack projects has grown steadily. We have the
> > following:
> >
> >- Mitaka
> >   - neutron: 1.40GB
> >   - nova: 1.70GB
> >   - swift: 640MB
> >   - cinder: 730MB
> >   - keystone: 760MB
> >   - horizon: 17MB
> >   - glance: 538MB
> >- Newton
> >- neutron: 1.59GB (+13%)
> >   - nova: 1.67GB (-1%)
> >   - swift: 779MB (+21%)
> >   - cinder: 878MB (+20%)
> >   - keystone: 919MB (+20%)
> >   - horizon: 21MB (+23%)
> >   - glance: 721MB (+34%)
> >- Ocata
> >   - neutron: 1.75GB (+10%)
> >   - nova: 1.95GB (%16%)
> >   - swift: 703MB (-9%)
> >   - cinder: 920MB (4%)
> >   - keystone: 903MB (-1%)
> >   - horizon: 25MB (+20%)
> >   - glance: 740MB (+2%)
> >
> > Numbers are approximated and I only took a couple of samples, but in a
> > nutshell, the majority of the services have seen double digit growth over
> > the past two cycles in terms of the amount or RSS memory they use.
> >
> > Since [1] is observed only since ocata [2], I imagine that's pretty
> > reasonable to assume that memory increase might as well be a determining
> > factor to the oom-kills we see in the gate.
> >
> > Profiling and surgically reducing the memory used by each component in
> > each service is a lengthy process, but I'd rather see some gate relief
> > right away. Reducing the number of API workers helps bring the RSS memory
> > down back to mitaka levels:
> >
> >- neutron: 1.54GB
> >- nova: 1.24GB
> >- swift: 694MB
> >- cinder: 778MB
> >- keystone: 891MB
> >- horizon: 24MB
> >- glance: 490MB
> >
> > However, it may have other side effects, like longer execution times, or
> > increase of timeouts.
> >
> > Where do we go from here? I am not particularly fond of stop-gap [4], but
> > it is the one fix that most widely address the memory increase we have
> > experienced across the board.
> >
> > Thanks,
> > Armando
> >
> > [1] https://bugs.launchpad.net/neutron/+bug/1656386
> > [2] http://logstash.openstack.org/#/dashboard/file/logstash.
> > json?query=message:%5C%22oom-killer%5C%22%20AND%20tags:syslog
> > [3] http://logs.openstack.org/21/427921/1/check/gate-
> > tempest-dsvm-neutron-full-ubuntu-xenial/82084c2/
> > [4] https://review.openstack.org/#/c/427921
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> [1.2  ]
> [2  ]
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Neutron team social event in Barcelona

[IWAMOTO Toshihiro]

+1

Thanks for organizing.

At Fri, 14 Oct 2016 13:30:57 -0500,
Miguel Lavalle wrote:
> 
> [1  ]
> [1.1  ]
> Dear Neutrinos,
> 
> I am organizing a social event for the team on Thursday 27th at 19:30.
> After doing some Google research, I am proposing Raco de la Vila, which is
> located in Poblenou: http://www.racodelavila.com/en/index.htm. The menu is
> here: http://www.racodelavila.com/en/carta-racodelavila.htm
> 
> It is easy to get there by subway from the Summit venue:
> https://goo.gl/maps/HjaTEcBbDUR2. I made a reservation for 25 people under
> 'Neutron' or "Miguel Lavalle". Please confirm your attendance so we can get
> a final count.
> 
> Here's some reviews:
> https://www.tripadvisor.com/Restaurant_Review-g187497-d1682057-Reviews-Raco_De_La_Vila-Barcelona_Catalonia.html
> 
> Cheers

--
IWAMOTO Toshihiro

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] OVSDB native interface as default in gate jobs

[IWAMOTO Toshihiro]

At Wed, 08 Jun 2016 03:26:59 -0700,
Clark Boylan wrote:
> 
> On Tue, Apr 5, 2016, at 08:32 PM, IWAMOTO Toshihiro wrote:
> > At Tue, 5 Apr 2016 12:57:33 -0400,
> > Assaf Muller wrote:
> > > 
> > > On Tue, Apr 5, 2016 at 12:35 PM, Sean M. Collins <s...@coreitpro.com> 
> > > wrote:
> > > > Russell Bryant wrote:
> > > >> because they are related to two different command line utilities
> > > >> (ovs-vsctl vs ovs-ofctl) that speak two different protocols (OVSDB vs
> > > >> OpenFlow) that talk to two different daemons on the system 
> > > >> (ovsdb-server vs
> > > >> ovs-vswitchd) ?
> > > >
> > > > True, they influence two different daemons - but it's really two options
> > > > that both have two settings:
> > > >
> > > > * "talk to it via the CLI tool"
> > > > * "talk to it via a native interface"
> > > >
> > > > How likely is it to have one talking via native interface and the other
> > > > via CLI?
> > > 
> > > The ovsdb native interface is a couple of cycles more mature than the
> > > openflow one, I see how some users would use one but not the other.
> > 
> > The native of_interface has been tested by periodic jobs and seems
> > pretty stable.
> > 
> > http://graphite.openstack.org/dashboard/#neutron-ovs-native
> > 
> > > > Also, if the native interface is faster, I think we should consider
> > > > making it the default.
> > > 
> > > Definitely. I'd prefer to deprecate and delete the cli interfaces and
> > > keep only the native interfaces in the long run.
> > > 
> > > >
> > > > --
> > > > Sean M. Collins
> > 
> > The native of_interface is definitely faster than the CLI alternative,
> > but (un?)fortunately that's not a performance bottleneck.
> > 
> > The transition would be a gain, but it comes with uncovering a few
> > unidentified bugs etc.
> > 
> > Anyway, I'll post an updated version of performance comparison shortly.
> 
> Going to resurrect this thread to see where we have gotten. Did an
> updated comparison ever get posted? If so I mised it. Looks like Neutron
> does have functional tests now that use the native interfaces for ovsdb
> and openflow. The change to update the default interface for ovsdb is
> still in review though.
> 

It happened on a different mail thread, and I haven't done any
comparison with ovsdb drivers.

http://lists.openstack.org/pipermail/openstack-dev/2016-April/091639.html
http://lists.openstack.org/pipermail/openstack-dev/2016-April/091642.html



> Any chance that we document these choices yet?

I've put a patch that switches to the native of_interface.
If we cannot switch the default, better documentation might help.

> Clark
> 

--
IWAMOTO Toshihiro

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] work on Common Flow Classifier and OVS Agent extension for Newton cycle

[IWAMOTO Toshihiro]

At Wed, 20 Apr 2016 14:12:07 +0200,
Miguel Angel Ajo Pelayo wrote:
> 
> I think this is an interesting topic.
> 
> What do you mean exactly by FC ? (feature chaining?)
> 
> I believe we have three things to look at:  (sorry for the TL)
> 
> 1) The generalization of traffic filters / traffic classifiers. Having
> common models, some sort of common API or common API structure
> available, and translators to convert those filters to iptables,
> openflow filters, etc..
> 
> 2) The enhancement of extensiblity of agents via Extension API.
> 
> 3) How we chain features in OpenFlow, which current approach of just
> inserting rules, renders into incompatible extensions. This becomes
> specially relevant for the new openvswitch firewall.
> 
> 2 and 3 are interlinked, and a good mechanism to enhance (3) should be
> provided in (2).
> 
> We need to resolve:
> 
> a) The order of tables, and how openflow actions chain the
> different features in the pipeline.  Some naive thinking brings me
> into the idea that we need to identify different input/output stages
> of packet processing, and every feature/extension declares the point
> where it needs to be. And then when we have all features, every
> feature get's it's own table number, and the "next" action in
> pipeline.

Can we create an API that allocates flow insertion points and table
numbers?  How can we ensure correct ordering of flows?
IMHO, it might be a time to collect low-level flow operation functions
into a single repository and test interoperability there.

> b) We need to have a way to request openflow registers to use in
> extensions, so one extension doesn't overwrite other's registers
> 
>c) Those registers need to be given a logical names that other
> extensions can query for (for example "port_number", "local_zone",
> etc..) , and those standard registers should be filled in for all
> extensions at the input stage.
> 
>and probably c,d,e,f,g,h what I didn't manage to think of.
> 
> On Fri, Apr 15, 2016 at 11:13 PM, Cathy Zhang  
> wrote:
> > Hi Reedip,
> >
> >
> >
> > Sure will include you in the discussion. Let me know if there are other
> > Tap-as-a-Service members who would like to join this initiative.
> >
> >
> >
> > Cathy
> >
> >
> >
> > From: reedip banerjee [mailto:reedi...@gmail.com]
> > Sent: Thursday, April 14, 2016 7:03 PM
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: Re: [openstack-dev] [neutron] work on Common Flow Classifier and
> > OVS Agent extension for Newton cycle
> >
> >
> >
> > Speaking on behalf of Tap-as-a-Service members, we would also be very much
> > interested in the following initiative :)
> >
> >
> >
> > On Fri, Apr 15, 2016 at 5:14 AM, Ihar Hrachyshka 
> > wrote:
> >
> > Cathy Zhang  wrote:
> >
> >
> > I think there is no formal spec or anything, just some emails around there.
> >
> > That said, I don’t follow why it’s a requirement for SFC to switch to l2
> > agent extension mechanism. Even today, with SFC maintaining its own agent,
> > there are no clear guarantees for flow priorities that would avoid all
> > possible conflicts.
> >
> > Cathy> There is no requirement for SFC to switch. My understanding is that
> > current L2 agent extension does not solve the conflicting entry issue if two
> > features inject the same priority table entry. I think this new L2 agent
> > effort is try to come up with a mechanism to resolve this issue. Of course
> > if each feature( SFC or Qos) uses its own agent, then there is no
> > coordination and no way to avoid conflicts.
> >
> >
> > Sorry, I probably used misleading wording. I meant, why do we consider the
> > semantic flow management support in l2 agent extension framework a
> > *prerequisite* for SFC to switch to l2 agent extensions? The existing
> > framework should already allow SFC to achieve what you have in the
> > subproject tree implemented as a separate agent (essentially a fork of OVS
> > agent). It will also set SFC to use standard extension mechanisms instead of
> > hacky inheritance from OVS agent classes. So even without the strict
> > semantic flow management, there is benefit for the subproject.
> >
> > With that in mind, I would split this job into 3 pieces:
> > * first, adopt l2 agent extension mechanism for SFC functionality (dropping
> > custom agent);
> > * then, work on semantic flow management support in OVS agent API class [1];
> > * once the feature emerges, switch SFC l2 agent extension to the new
> > framework to manage SFC flows.
> >
> > I would at least prioritize the first point and target it to Newton-1. Other
> > bullet points may take significant time to bake.
> >
> > [1]
> > https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_agent_extension_api.py
> >
> >
> >
> > Ihar
> >
> > __
> > OpenStack 

Re: [openstack-dev] [Neutron] OVS flow modification performance

[IWAMOTO Toshihiro]

At Mon, 11 Apr 2016 14:42:59 +0200,
Miguel Angel Ajo Pelayo wrote:
> 
> On Mon, Apr 11, 2016 at 11:40 AM, IWAMOTO Toshihiro
> <iwam...@valinux.co.jp> wrote:
> > At Fri, 8 Apr 2016 12:21:21 +0200,
> > Miguel Angel Ajo Pelayo wrote:
> >>
> >> Hi, good that you're looking at this,
> >>
> >>
> >> You could create a lot of ports with this method [1] and a bit of extra
> >> bash, without the extra expense of instance RAM.
> >>
> >>
> >> [1]
> >> http://www.ajo.es/post/89207996034/creating-a-network-interface-to-tenant-network-in
> >>
> >>
> >> This effort is going to be still more relevant in the context of
> >> openvswitch firewall. We still need to make sure it's tested with the
> >> native interface, and eventually we will need flow bundling (like in
> >> ovs-ofctl --bundle add-flows) where the whole addition/removal/modification
> >> is sent to be executed atomically by the switch.
> >
> > Bad news is that ovs-firewall isn't currently using the native
> > of_interface much.  I can add install_xxx methods to
> > OpenFlowSwitchMixin classes so that ovs-firewall can use the native
> > interface.
> > Do you have a plan for implementing flow bundling or using conjunction?
> >
> 
> Adding Jakub to the thread,
> 
> IMO, if the native interface is going to provide us with greater speed
> for rule manipulation, we should look into it.
> 
> We don't use bundling or conjunctions yet, but it's part of the plan.
> Bundling will allow atomicity of operations with rules (switching
> firewall rules, etc, as we have with iptables-save /
> iptables-restore), and conjunctions will reduce the number of entries.
> (No expansion of IP addresses for remote groups, no expansion of
> security group rules per port, when several ports are on the same
> security group on the same compute host).
> 
> Do we have any metric of bare rule manipulation time (ms/rule, for example)?

No bare numbers but from a graph in the other mail I sent last week,
bind_devices for 160 ports (iirc, that amounts to 800 flows) takes
4.5sec with of_interface=native and 8sec with of_interface=ovs-ofctl,
which means an native add-flow is 4ms faster than the other.

As the ovs firewall uses DeferredOVSBridge and has less exec
overheads, I have no idea how much gain the native of_interface
brings.

> As a note, we're around 80 rules/port with IPv6 + IPv4 on the default
> sec group plus a couple of rules.

I booted 120VMs on one network and the default security group
generated 62k flows.  It seems using conjunction is the #1 item for
performance.



> 
> >> On Thu, Apr 7, 2016 at 10:00 AM, IWAMOTO Toshihiro <iwam...@valinux.co.jp>
> >> wrote:
> >>
> >> > At Thu, 07 Apr 2016 16:33:02 +0900,
> >> > IWAMOTO Toshihiro wrote:
> >> > >
> >> > > At Mon, 18 Jan 2016 12:12:28 +0900,
> >> > > IWAMOTO Toshihiro wrote:
> >> > > >
> >> > > > I'm sending out this mail to share the finding and discuss how to
> >> > > > improve with those interested in neutron ovs performance.
> >> > > >
> >> > > > TL;DR: The native of_interface code, which has been merged recently
> >> > > > and isn't default, seems to consume less CPU time but gives a mixed
> >> > > > result.  I'm looking into this for improvement.
> >> > >
> >> > > I went on to look at implementation details of eventlet etc, but it
> >> > > turned out to be fairly simple.  The OVS agent in the
> >> > > of_interface=native mode waits for a openflow connection from
> >> > > ovs-vswitchd, which can take up to 5 seconds.
> >> > >
> >> > > Please look at the attached graph.
> >> > > The x-axis is time from agent restarts, the y-axis is numbers of ports
> >> > > processed (in treat_devices and bind_devices).  Each port is counted
> >> > > twice; the first slope is treat_devices and the second is
> >> > > bind_devices.  The native of_interface needs some more time on
> >> > > start-up, but bind_devices is about 2x faster.
> >> > >
> >> > > The data was collected with 160 VMs with the devstack default settings.
> >> >
> >> > And if you wonder how other services are doing meanwhile, here is a
> >> > bonus chart.
> >> >
> >> > The ovs agent was restarted 3 times with of_interface=native, then 3
> >> > times with of_interface=ovs-ofctl.
> >> >
> &

Re: [openstack-dev] [Neutron] OVS flow modification performance

[IWAMOTO Toshihiro]

At Fri, 8 Apr 2016 12:21:21 +0200,
Miguel Angel Ajo Pelayo wrote:
> 
> Hi, good that you're looking at this,
> 
> 
> You could create a lot of ports with this method [1] and a bit of extra
> bash, without the extra expense of instance RAM.
> 
> 
> [1]
> http://www.ajo.es/post/89207996034/creating-a-network-interface-to-tenant-network-in
> 
> 
> This effort is going to be still more relevant in the context of
> openvswitch firewall. We still need to make sure it's tested with the
> native interface, and eventually we will need flow bundling (like in
> ovs-ofctl --bundle add-flows) where the whole addition/removal/modification
> is sent to be executed atomically by the switch.

Bad news is that ovs-firewall isn't currently using the native
of_interface much.  I can add install_xxx methods to
OpenFlowSwitchMixin classes so that ovs-firewall can use the native
interface.
Do you have a plan for implementing flow bundling or using conjunction?

> On Thu, Apr 7, 2016 at 10:00 AM, IWAMOTO Toshihiro <iwam...@valinux.co.jp>
> wrote:
> 
> > At Thu, 07 Apr 2016 16:33:02 +0900,
> > IWAMOTO Toshihiro wrote:
> > >
> > > At Mon, 18 Jan 2016 12:12:28 +0900,
> > > IWAMOTO Toshihiro wrote:
> > > >
> > > > I'm sending out this mail to share the finding and discuss how to
> > > > improve with those interested in neutron ovs performance.
> > > >
> > > > TL;DR: The native of_interface code, which has been merged recently
> > > > and isn't default, seems to consume less CPU time but gives a mixed
> > > > result.  I'm looking into this for improvement.
> > >
> > > I went on to look at implementation details of eventlet etc, but it
> > > turned out to be fairly simple.  The OVS agent in the
> > > of_interface=native mode waits for a openflow connection from
> > > ovs-vswitchd, which can take up to 5 seconds.
> > >
> > > Please look at the attached graph.
> > > The x-axis is time from agent restarts, the y-axis is numbers of ports
> > > processed (in treat_devices and bind_devices).  Each port is counted
> > > twice; the first slope is treat_devices and the second is
> > > bind_devices.  The native of_interface needs some more time on
> > > start-up, but bind_devices is about 2x faster.
> > >
> > > The data was collected with 160 VMs with the devstack default settings.
> >
> > And if you wonder how other services are doing meanwhile, here is a
> > bonus chart.
> >
> > The ovs agent was restarted 3 times with of_interface=native, then 3
> > times with of_interface=ovs-ofctl.
> >
> > As the test machine has 16 CPUs, 6.25% CPU usage can mean a single
> > threaded process is CPU bound.
> >
> > Frankly, the OVS agent would have little rooms for improvement than
> > other services.  Also, it might be fun to draw similar charts for
> > other types of workloads.
> >
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] OVS flow modification performance

[IWAMOTO Toshihiro]

At Thu, 07 Apr 2016 16:33:02 +0900,
IWAMOTO Toshihiro wrote:
> 
> At Mon, 18 Jan 2016 12:12:28 +0900,
> IWAMOTO Toshihiro wrote:
> > 
> > I'm sending out this mail to share the finding and discuss how to
> > improve with those interested in neutron ovs performance.
> > 
> > TL;DR: The native of_interface code, which has been merged recently
> > and isn't default, seems to consume less CPU time but gives a mixed
> > result.  I'm looking into this for improvement.
> 
> I went on to look at implementation details of eventlet etc, but it
> turned out to be fairly simple.  The OVS agent in the
> of_interface=native mode waits for a openflow connection from
> ovs-vswitchd, which can take up to 5 seconds.
> 
> Please look at the attached graph.
> The x-axis is time from agent restarts, the y-axis is numbers of ports
> processed (in treat_devices and bind_devices).  Each port is counted
> twice; the first slope is treat_devices and the second is
> bind_devices.  The native of_interface needs some more time on
> start-up, but bind_devices is about 2x faster.
> 
> The data was collected with 160 VMs with the devstack default settings.

And if you wonder how other services are doing meanwhile, here is a
bonus chart.

The ovs agent was restarted 3 times with of_interface=native, then 3
times with of_interface=ovs-ofctl.

As the test machine has 16 CPUs, 6.25% CPU usage can mean a single
threaded process is CPU bound.

Frankly, the OVS agent would have little rooms for improvement than
other services.  Also, it might be fun to draw similar charts for
other types of workloads.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] OVS flow modification performance

[IWAMOTO Toshihiro]

At Mon, 18 Jan 2016 12:12:28 +0900,
IWAMOTO Toshihiro wrote:
> 
> I'm sending out this mail to share the finding and discuss how to
> improve with those interested in neutron ovs performance.
> 
> TL;DR: The native of_interface code, which has been merged recently
> and isn't default, seems to consume less CPU time but gives a mixed
> result.  I'm looking into this for improvement.

I went on to look at implementation details of eventlet etc, but it
turned out to be fairly simple.  The OVS agent in the
of_interface=native mode waits for a openflow connection from
ovs-vswitchd, which can take up to 5 seconds.

Please look at the attached graph.
The x-axis is time from agent restarts, the y-axis is numbers of ports
processed (in treat_devices and bind_devices).  Each port is counted
twice; the first slope is treat_devices and the second is
bind_devices.  The native of_interface needs some more time on
start-up, but bind_devices is about 2x faster.

The data was collected with 160 VMs with the devstack default settings.

> * Introduction
> 
> With an ML2+ovs Neutron configuration, openflow rule modification
> happens often and is somewhat a heavy operation as it involves
> exec() of the ovs-ofctl command.
> 
> The native of_interface driver doesn't use the ovs-ofctl command and
> should have less performance impact on the system.  This document
> tries to confirm this hypothesis.
> 
> 
> * Method
> 
> In order to focus on openflow rule operation time and avoid noise from
> other operations (VM boot-up, etc.), neutron-openvswitch-agent was
> restarted and the time it took to reconfigure the flows was measured.
> 
> 1. Use devstack to start a test environment.  As debug logs generate
>considable amount of load, ENABLE_DEBUG_LOG_LEVEL was set to false.
> 2. Apply https://review.openstack.org/#/c/267905/ to enable
>measurement of flow reconfiguration times.
> 3. Boot 80 m1.nano instances.  In my setup, this generates 404 br-int
>flows.  If you have >16G RAM, more could be booted.
> 4. Stop neutron-openvswitch-agent and restart with --run-once arg.
>Use time, oprofile, and python's cProfile (use --profile arg) to
>collect data.
> 
> * Results
> 
> Execution time (averages of 3 runs):
> 
> native 28.3s user 2.9s sys 0.4s
> ovs-ofctl  25.7s user 2.2s sys 0.3s
> 
> ovs-ofctl runs faster and seems to use less CPU, but the above doesn't
> count in execution time of ovs-ofctl.

With 160 VMs and debug=false for the OVS agent and the neutron-server,

Execution time (averages and SDs of 10 runs):

native 56.4+-3.4s  user 8.7+-0.1s   sys 0.82+-0.04s
ovs-ofctl  55.9+-1.0s  user 6.9+-0.08s  sys 0.67+-0.05s

To exclude the openflow connection waits,
times between log outputs of "Loaded agent extensions" and
"Configuration for devices up completed" is also compared:

native 48.2+-0.49s
ovs-ofctl  53.2+-0.99s

The native of_interface is the clear winner.

> Oprofile data collected by running "operf -s -t" contain the
> information.
> 
> With of_interface=native config, "opreport tgid:" shows:
> 
>samples|  %|
> --
> 87408 100.000 python2.7
>   CPU_CLK_UNHALT...|
> samples|  %|
>   --
>   69160 79.1232 python2.7
>8416  9.6284 vmlinux-3.13.0-24-generic
> 
> and "opreport --merge tgid" doesn't show ovs-ofctl.
> 
> With of_interface=ovs-ofctl, "opreport tgid:" shows:
> 
>samples|  %|
> --
> 62771 100.000 python2.7
> CPU_CLK_UNHALT...|
>   samples|  %|
> --
> 49418 78.7274 python2.7
>  6483 10.3280 vmlinux-3.13.0-24-generic
> 
> and  "opreport --merge tgid" shows CPU consumption by ovs-ofctl 
> 
> 35774  3.5979 ovs-ofctl
> CPU_CLK_UNHALT...|
>   samples|  %|
> --
> 28219 78.8813 vmlinux-3.13.0-24-generic
>  3487  9.7473 ld-2.19.so
>  2301  6.4320 ovs-ofctl
> 
> Comparing 87408 (native python) with 62771+35774, the native
> of_interface uses 0.4s less CPU time overall.
> 
> * Conclusion and future steps
> 
> The native of_interface uses slightly less CPU time but takes longer
> time to complete a flow reconfiguration after an agent restart.
> 
> As an OVS agent accounts for only 1/10th of total CPU usage during a
> flow reconfiguration (data not shown), there may be other areas for
> improvement.
> 
> The cProfile Python module gives more fine grained data, but no
> apparent performance bottleneck was found.  The

Re: [openstack-dev] [neutron] OVSDB native interface as default in gate jobs

[IWAMOTO Toshihiro]

At Tue, 5 Apr 2016 12:57:33 -0400,
Assaf Muller wrote:
> 
> On Tue, Apr 5, 2016 at 12:35 PM, Sean M. Collins <s...@coreitpro.com> wrote:
> > Russell Bryant wrote:
> >> because they are related to two different command line utilities
> >> (ovs-vsctl vs ovs-ofctl) that speak two different protocols (OVSDB vs
> >> OpenFlow) that talk to two different daemons on the system (ovsdb-server vs
> >> ovs-vswitchd) ?
> >
> > True, they influence two different daemons - but it's really two options
> > that both have two settings:
> >
> > * "talk to it via the CLI tool"
> > * "talk to it via a native interface"
> >
> > How likely is it to have one talking via native interface and the other
> > via CLI?
> 
> The ovsdb native interface is a couple of cycles more mature than the
> openflow one, I see how some users would use one but not the other.

The native of_interface has been tested by periodic jobs and seems
pretty stable.

http://graphite.openstack.org/dashboard/#neutron-ovs-native

> > Also, if the native interface is faster, I think we should consider
> > making it the default.
> 
> Definitely. I'd prefer to deprecate and delete the cli interfaces and
> keep only the native interfaces in the long run.
> 
> >
> > --
> > Sean M. Collins

The native of_interface is definitely faster than the CLI alternative,
but (un?)fortunately that's not a performance bottleneck.

The transition would be a gain, but it comes with uncovering a few
unidentified bugs etc.

Anyway, I'll post an updated version of performance comparison shortly.

--
IWAMOTO Toshihiro

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] OVS flow modification performance

[IWAMOTO Toshihiro]

At Sat, 30 Jan 2016 02:08:55 +,
Wuhongning wrote:
> 
> By our testing, ryu openflow has greatly improved the performance, with 500 
> port vxlan flow table, from 15s to 2.5s, 6 times better.

That's quite a impressive number.
What tests did you do?  Could you share some details?

Also, although unlikely, but please make sure your measurements aren't
affected by https://bugs.launchpad.net/neutron/+bug/1538368 .


> ____
> From: IWAMOTO Toshihiro [iwam...@valinux.co.jp]
> Sent: Monday, January 25, 2016 5:08 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [Neutron] OVS flow modification performance
> 
> At Thu, 21 Jan 2016 02:59:16 +,
> Wuhongning wrote:
> >
> > I don't think 400 flows can show the difference , do you have setup any 
> > tunnel peer?
> >
> > In fact we may set the network type as "vxlan", then make a fake MD 
> > simulate sending l2pop fdb add messages, to push ten's of thousands flows 
> > into the testing ovs agent.
> 
> I chose this method because I didn't want to write such extra code for
> measurements. ;)
> Of course, I'd love to see data from other test environments and other
> workload than agent restarts.
> 
> Also, we now have https://review.openstack.org/#/c/271939/ and can
> profile neutron-server (and probably others, too).
> I couldn't find non-trivial findings until now, though.
> 
> > 
> > From: IWAMOTO Toshihiro [iwam...@valinux.co.jp]
> > Sent: Monday, January 18, 2016 4:37 PM
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: Re: [openstack-dev] [Neutron] OVS flow modification performance
> >
> > At Mon, 18 Jan 2016 00:42:32 -0500,
> > Kevin Benton wrote:
> > >
> > > Thanks for doing this. A couple of questions:
> > >
> > > What were your rootwrap settings when running these tests? Did you just
> > > have it calling sudo directly?
> >
> > I used devstack's default, which runs root_helper_daemon.
> >
> > > Also, you mention that this is only ~10% of the time spent during flow
> > > reconfiguration. What other areas are eating up so much time?
> >
> >
> > In another run,
> >
> > $ for f in `cat tgidlist.n2`; do echo -n $f; opreport -n tgid:$f --merge 
> > tid|head -1|tr -d '\n'; (cd bg; opreport -n tgid:$f --merge tid|head 
> > -1);echo; done|sort -nr -k +2
> > 10071   239058 100.000 python2.714922 100.000 python2.7
> > 999592328 100.000 python2.711450 100.000 python2.7
> > 757988202 100.000 python2.7(18596)
> > 1109451560 100.000 python2.747964 100.000 python2.7
> > 703549687 100.000 python2.740678 100.000 python2.7
> > 1109349380 100.000 python2.736004 100.000 python2.7
> > (legend:
> >  )
> >
> > These processes are neutron-server, nova-api,
> > neutron-openvswitch-agent, nova-conductor, dstat and nova-conductor in
> > a decending order.
> >
> > So neutron-server uses about 3x CPU time than the ovs agent,
> > nova-api's CPU usage is similar to the ovs agent's, and the others
> > aren't probably significant.
> >
> > > Cheers,
> > > Kevin Benton
> > >
> > > On Sun, Jan 17, 2016 at 10:12 PM, IWAMOTO Toshihiro 
> > > <iwam...@valinux.co.jp>
> > > wrote:
> > >
> > > > I'm sending out this mail to share the finding and discuss how to
> > > > improve with those interested in neutron ovs performance.
> > > >
> > > > TL;DR: The native of_interface code, which has been merged recently
> > > > and isn't default, seems to consume less CPU time but gives a mixed
> > > > result.  I'm looking into this for improvement.
> > > >
> > > > * Introduction
> > > >
> > > > With an ML2+ovs Neutron configuration, openflow rule modification
> > > > happens often and is somewhat a heavy operation as it involves
> > > > exec() of the ovs-ofctl command.
> > > >
> > > > The native of_interface driver doesn't use the ovs-ofctl command and
> > > > should have less performance impact on the system.  This document
> > > > tries to confirm this hypothesis.
> > > >
> > > >
> > > > * Method
> > > >
> > > > In order to focus on openflow rule operation time and avoid noise from
> > > > other operations (VM boot-up, etc.), neutron-openvswitch-agent was
> > > > restarted a

Re: [openstack-dev] [Neutron] OVS flow modification performance

[IWAMOTO Toshihiro]

At Thu, 21 Jan 2016 02:59:16 +,
Wuhongning wrote:
> 
> I don't think 400 flows can show the difference , do you have setup any 
> tunnel peer?
> 
> In fact we may set the network type as "vxlan", then make a fake MD simulate 
> sending l2pop fdb add messages, to push ten's of thousands flows into the 
> testing ovs agent.

I chose this method because I didn't want to write such extra code for
measurements. ;)
Of course, I'd love to see data from other test environments and other
workload than agent restarts.

Also, we now have https://review.openstack.org/#/c/271939/ and can
profile neutron-server (and probably others, too).
I couldn't find non-trivial findings until now, though.

> ________
> From: IWAMOTO Toshihiro [iwam...@valinux.co.jp]
> Sent: Monday, January 18, 2016 4:37 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [Neutron] OVS flow modification performance
> 
> At Mon, 18 Jan 2016 00:42:32 -0500,
> Kevin Benton wrote:
> >
> > Thanks for doing this. A couple of questions:
> >
> > What were your rootwrap settings when running these tests? Did you just
> > have it calling sudo directly?
> 
> I used devstack's default, which runs root_helper_daemon.
> 
> > Also, you mention that this is only ~10% of the time spent during flow
> > reconfiguration. What other areas are eating up so much time?
> 
> 
> In another run,
> 
> $ for f in `cat tgidlist.n2`; do echo -n $f; opreport -n tgid:$f --merge 
> tid|head -1|tr -d '\n'; (cd bg; opreport -n tgid:$f --merge tid|head 
> -1);echo; done|sort -nr -k +2
> 10071   239058 100.000 python2.714922 100.000 python2.7
> 999592328 100.000 python2.711450 100.000 python2.7
> 757988202 100.000 python2.7(18596)
> 1109451560 100.000 python2.747964 100.000 python2.7
> 703549687 100.000 python2.740678 100.000 python2.7
> 1109349380 100.000 python2.736004 100.000 python2.7
> (legend:
>  )
> 
> These processes are neutron-server, nova-api,
> neutron-openvswitch-agent, nova-conductor, dstat and nova-conductor in
> a decending order.
> 
> So neutron-server uses about 3x CPU time than the ovs agent,
> nova-api's CPU usage is similar to the ovs agent's, and the others
> aren't probably significant.
> 
> > Cheers,
> > Kevin Benton
> >
> > On Sun, Jan 17, 2016 at 10:12 PM, IWAMOTO Toshihiro <iwam...@valinux.co.jp>
> > wrote:
> >
> > > I'm sending out this mail to share the finding and discuss how to
> > > improve with those interested in neutron ovs performance.
> > >
> > > TL;DR: The native of_interface code, which has been merged recently
> > > and isn't default, seems to consume less CPU time but gives a mixed
> > > result.  I'm looking into this for improvement.
> > >
> > > * Introduction
> > >
> > > With an ML2+ovs Neutron configuration, openflow rule modification
> > > happens often and is somewhat a heavy operation as it involves
> > > exec() of the ovs-ofctl command.
> > >
> > > The native of_interface driver doesn't use the ovs-ofctl command and
> > > should have less performance impact on the system.  This document
> > > tries to confirm this hypothesis.
> > >
> > >
> > > * Method
> > >
> > > In order to focus on openflow rule operation time and avoid noise from
> > > other operations (VM boot-up, etc.), neutron-openvswitch-agent was
> > > restarted and the time it took to reconfigure the flows was measured.
> > >
> > > 1. Use devstack to start a test environment.  As debug logs generate
> > >considable amount of load, ENABLE_DEBUG_LOG_LEVEL was set to false.
> > > 2. Apply https://review.openstack.org/#/c/267905/ to enable
> > >measurement of flow reconfiguration times.
> > > 3. Boot 80 m1.nano instances.  In my setup, this generates 404 br-int
> > >flows.  If you have >16G RAM, more could be booted.
> > > 4. Stop neutron-openvswitch-agent and restart with --run-once arg.
> > >Use time, oprofile, and python's cProfile (use --profile arg) to
> > >collect data.
> > >
> > > * Results
> > >
> > > Execution time (averages of 3 runs):
> > >
> > > native 28.3s user 2.9s sys 0.4s
> > > ovs-ofctl  25.7s user 2.2s sys 0.3s
> > >
> > > ovs-ofctl runs faster and seems to use less CPU, but the above doesn't
> > > count in execution time of ovs-ofctl.
> > >
> > > Oprofile dat

Re: [openstack-dev] [Neutron] OVS flow modification performance

[IWAMOTO Toshihiro]

At Mon, 18 Jan 2016 00:42:32 -0500,
Kevin Benton wrote:
> 
> Thanks for doing this. A couple of questions:
> 
> What were your rootwrap settings when running these tests? Did you just
> have it calling sudo directly?

I used devstack's default, which runs root_helper_daemon.

> Also, you mention that this is only ~10% of the time spent during flow
> reconfiguration. What other areas are eating up so much time?


In another run,

$ for f in `cat tgidlist.n2`; do echo -n $f; opreport -n tgid:$f --merge 
tid|head -1|tr -d '\n'; (cd bg; opreport -n tgid:$f --merge tid|head -1);echo; 
done|sort -nr -k +2
10071   239058 100.000 python2.714922 100.000 python2.7 
999592328 100.000 python2.711450 100.000 python2.7 
757988202 100.000 python2.7(18596)
1109451560 100.000 python2.747964 100.000 python2.7 
703549687 100.000 python2.740678 100.000 python2.7 
1109349380 100.000 python2.736004 100.000 python2.7 
(legend:
 )

These processes are neutron-server, nova-api,
neutron-openvswitch-agent, nova-conductor, dstat and nova-conductor in
a decending order.

So neutron-server uses about 3x CPU time than the ovs agent,
nova-api's CPU usage is similar to the ovs agent's, and the others
aren't probably significant.

> Cheers,
> Kevin Benton
> 
> On Sun, Jan 17, 2016 at 10:12 PM, IWAMOTO Toshihiro <iwam...@valinux.co.jp>
> wrote:
> 
> > I'm sending out this mail to share the finding and discuss how to
> > improve with those interested in neutron ovs performance.
> >
> > TL;DR: The native of_interface code, which has been merged recently
> > and isn't default, seems to consume less CPU time but gives a mixed
> > result.  I'm looking into this for improvement.
> >
> > * Introduction
> >
> > With an ML2+ovs Neutron configuration, openflow rule modification
> > happens often and is somewhat a heavy operation as it involves
> > exec() of the ovs-ofctl command.
> >
> > The native of_interface driver doesn't use the ovs-ofctl command and
> > should have less performance impact on the system.  This document
> > tries to confirm this hypothesis.
> >
> >
> > * Method
> >
> > In order to focus on openflow rule operation time and avoid noise from
> > other operations (VM boot-up, etc.), neutron-openvswitch-agent was
> > restarted and the time it took to reconfigure the flows was measured.
> >
> > 1. Use devstack to start a test environment.  As debug logs generate
> >considable amount of load, ENABLE_DEBUG_LOG_LEVEL was set to false.
> > 2. Apply https://review.openstack.org/#/c/267905/ to enable
> >measurement of flow reconfiguration times.
> > 3. Boot 80 m1.nano instances.  In my setup, this generates 404 br-int
> >flows.  If you have >16G RAM, more could be booted.
> > 4. Stop neutron-openvswitch-agent and restart with --run-once arg.
> >Use time, oprofile, and python's cProfile (use --profile arg) to
> >collect data.
> >
> > * Results
> >
> > Execution time (averages of 3 runs):
> >
> > native 28.3s user 2.9s sys 0.4s
> > ovs-ofctl  25.7s user 2.2s sys 0.3s
> >
> > ovs-ofctl runs faster and seems to use less CPU, but the above doesn't
> > count in execution time of ovs-ofctl.
> >
> > Oprofile data collected by running "operf -s -t" contain the
> > information.
> >
> > With of_interface=native config, "opreport tgid:" shows:
> >
> >samples|  %|
> > --
> > 87408 100.000 python2.7
> > CPU_CLK_UNHALT...|
> >   samples|  %|
> > --
> > 69160 79.1232 python2.7
> >  8416  9.6284 vmlinux-3.13.0-24-generic
> >
> > and "opreport --merge tgid" doesn't show ovs-ofctl.
> >
> > With of_interface=ovs-ofctl, "opreport tgid:" shows:
> >
> >samples|  %|
> > --
> > 62771 100.000 python2.7
> > CPU_CLK_UNHALT...|
> >   samples|  %|
> > --
> > 49418 78.7274 python2.7
> >  6483 10.3280 vmlinux-3.13.0-24-generic
> >
> > and  "opreport --merge tgid" shows CPU consumption by ovs-ofctl
> >
> > 35774  3.5979 ovs-ofctl
> > CPU_CLK_UNHALT...|
> >   samples|  %|
> > --
> > 28219 78.8813 vmlinux-3.13.0-24-generic
> >  3487  9.7473 ld-2.19.so
> >  2301  6.4320 ovs-ofctl
> >
> > Comparing 87408 (native pytho

[openstack-dev] [Neutron] OVS flow modification performance

[IWAMOTO Toshihiro]

I'm sending out this mail to share the finding and discuss how to
improve with those interested in neutron ovs performance.

TL;DR: The native of_interface code, which has been merged recently
and isn't default, seems to consume less CPU time but gives a mixed
result.  I'm looking into this for improvement.

* Introduction

With an ML2+ovs Neutron configuration, openflow rule modification
happens often and is somewhat a heavy operation as it involves
exec() of the ovs-ofctl command.

The native of_interface driver doesn't use the ovs-ofctl command and
should have less performance impact on the system.  This document
tries to confirm this hypothesis.


* Method

In order to focus on openflow rule operation time and avoid noise from
other operations (VM boot-up, etc.), neutron-openvswitch-agent was
restarted and the time it took to reconfigure the flows was measured.

1. Use devstack to start a test environment.  As debug logs generate
   considable amount of load, ENABLE_DEBUG_LOG_LEVEL was set to false.
2. Apply https://review.openstack.org/#/c/267905/ to enable
   measurement of flow reconfiguration times.
3. Boot 80 m1.nano instances.  In my setup, this generates 404 br-int
   flows.  If you have >16G RAM, more could be booted.
4. Stop neutron-openvswitch-agent and restart with --run-once arg.
   Use time, oprofile, and python's cProfile (use --profile arg) to
   collect data.

* Results

Execution time (averages of 3 runs):

native 28.3s user 2.9s sys 0.4s
ovs-ofctl  25.7s user 2.2s sys 0.3s

ovs-ofctl runs faster and seems to use less CPU, but the above doesn't
count in execution time of ovs-ofctl.

Oprofile data collected by running "operf -s -t" contain the
information.

With of_interface=native config, "opreport tgid:" shows:

   samples|  %|
--
87408 100.000 python2.7
CPU_CLK_UNHALT...|
  samples|  %|
--
69160 79.1232 python2.7
 8416  9.6284 vmlinux-3.13.0-24-generic

and "opreport --merge tgid" doesn't show ovs-ofctl.

With of_interface=ovs-ofctl, "opreport tgid:" shows:

   samples|  %|
--
62771 100.000 python2.7
CPU_CLK_UNHALT...|
  samples|  %|
--
49418 78.7274 python2.7
 6483 10.3280 vmlinux-3.13.0-24-generic

and  "opreport --merge tgid" shows CPU consumption by ovs-ofctl 

35774  3.5979 ovs-ofctl
CPU_CLK_UNHALT...|
  samples|  %|
--
28219 78.8813 vmlinux-3.13.0-24-generic
 3487  9.7473 ld-2.19.so
 2301  6.4320 ovs-ofctl

Comparing 87408 (native python) with 62771+35774, the native
of_interface uses 0.4s less CPU time overall.

* Conclusion and future steps

The native of_interface uses slightly less CPU time but takes longer
time to complete a flow reconfiguration after an agent restart.

As an OVS agent accounts for only 1/10th of total CPU usage during a
flow reconfiguration (data not shown), there may be other areas for
improvement.

The cProfile Python module gives more fine grained data, but no
apparent performance bottleneck was found.  The data show more
eventlet context switches with the native of_interface, which is due
to how the native of_interface is written.  I'm looking into for
improving CPU usage and latency.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Team meeting on Tuesday 1400UTC

[IWAMOTO Toshihiro]

At Tue, 12 Jan 2016 17:28:19 -0600,
Doug Wiegley wrote:
> 
> I don’t think it ninja merged. It had plenty of reviews, and was open during 
> international hours. I don’t have any issue there.
> 
> I don’t like the crazy early meeting, so I set out to prove it didn’t matter:
> 
> Average attendance before rotating: 20.7 people
> Average attendance on Monday afternoons (U.S. time): 20.9
> Average attendance on Tuesday morning (U.S. time): 23.7
> 
> Stupid data, that’s not what I wanted to see.
> 
> I haven’t yet correlated people to which meeting time yet, but attendance was 
> slightly up during the crazy early hated time, across the 1.25 years it was 
> running (started 9/9/14). This is just people saying something; lurkers can 
> just read the logs.
> 
> Data is from eavesdrop meeting logs, if anyone else wants to crunch it.

Overall attendance isn't a great metric.  Actually, the population
does differ with meeting times.  Those who mostly attend only in one
of the two timeslots are:

$ python a.py
Sukhdev 6/23 17/22
ajo 22/23 4/22
obondarev 10/23 0/22
vikram 7/23 0/22
kevinbenton 9/23 22/22
ihrachyshka 13/23 1/22
Swami 0/23 8/22
(Legend:   )

The script is at bottom just in case if anyone cares. ;)

> Thanks,
> doug
> 
> 
> > On Jan 12, 2016, at 4:32 PM, Tony Breeds  wrote:
> > 
> > On Tue, Jan 12, 2016 at 01:27:30PM +0100, Ihar Hrachyshka wrote:
> >> Agreed with Gary on behalf of my European compatriots. (Note that I
> >> *personally* +1’d the patch because I don’t mind, doing late hours anyway;
> >> but it’s sad it was ninja merged without giving any chance for those from
> >> affected timezones to express their concerns).
> > 
> > So Ninja merged has a negative connotation that I refute.
> > 
> > I merged it.  It was judgment error, and I apologise for that.
> > 
> > * I found and read through the list thread.
> > * Saw only +1's yours included
> >- known you'd be affected I used your +1 as a barometer
> > 
> > My mistake was not noticing your request to leave the review open for 
> > longer.
> > 
> > I also noted in my review that reverting it is pretty low cost to back it 
> > out
> > if needed.
> > 
> > I understand that the 'root cause' for this change was the yaml2ical issue 
> > that
> > stemmed from having 2 odd week in a row.  We've fixed that [1]. I'm also
> > working a a more human concept of biweekly meeting in yaml2ical.
> > 
> > Tony
> > [1] the next time it could have been a problem is 2020/2021 ;P

#!/usr/bin/python
import os
import re

from scipy.stats import binom_test

# Run the following before executing this:
# $ wget http://eavesdrop.openstack.org/meetings/networking/2015/
# $ for f in `sed -n  '/[^g].txt"/ s/.*href=.\([^"]*\)*".*/\1/p' index.html ` ; 
do wget http://eavesdrop.openstack.org/meetings/networking/2015/$f; done

fname_re = re.compile('-(\d\d)\.\d\d\.txt$')
person_re = re.compile('^\* (.*[^_])_* \(\d+\)$')

count = {}
freq = {}
for f in os.listdir('.'):
m = fname_re.search(f)
if not m:
continue
hour = m.group(1)
if hour not in freq:
freq[hour] = {}
count[hour] = 0
count[hour] += 1
f1 = freq[hour]
with open(f) as ff:
while True:
l = ff.readline()
if l.startswith('People present'):
ll = ff.read().splitlines()
break
if not l:
raise Exception("Error reading %s" % f)
for l in ll:
m = person_re.match(l)
if m:
name = m.group(1)
f1[name] = f1.get(name, 0) + 1

assert set(freq.keys()) == set(['14', '21'])
people = set(freq['14'].keys()) | set(freq['21'].keys())
total = count['14'] + count['21']
c14 = count['14']
c21 = count['21']

sig_lvl = .05
for n in people:
f14 = freq['14'].get(n, 0)
f21 = freq['21'].get(n, 0)
p = float(f14 + f21) / total
if binom_test(f14, c14, p) < sig_lvl or binom_test(f21, c21, p) < sig_lvl:
print("%s %d/%d %d/%d" % (n, f14, c14, f21, c21))

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Neutron Social Meetup in Tokyo

[IWAMOTO Toshihiro]

At Mon, 26 Oct 2015 12:35:27 +0900,
Akihiro Motoki wrote:
> 
> Hi Antonio,
> 
> What you need to do is just to go.
> There is no need for sign up now.

And bring some cash, as per-person payment isn't available, I guess.

> 2015-10-26 4:17 GMT+09:00 Tonhão - Antonio Brasolin :
> > Hi. What do I have to do to go to this meeting? Do I have to sign up
> > somewhere? Thanks.
> >
> > On Fri, Oct 23, 2015 at 2:23 PM, Akihiro Motoki  wrote:
> >>
> >> Hi Neutron folks,
> >>
> >> We are pleased to announce Neutron social meet-up in Tokyo.
> >> Thanks Takashi and Hirofumi for the big help.
> >>
> >> I hope many of you will be there and enjoy the time.
> >> If you have made RSVP, don't miss it!
> >> We recommend  to join the beginning, but come and join us even if you
> >> arrive late.
> >>
> >> 
> >>
> >> Thursday, Oct 29 19:00-22:00
> >> Hokkaido (北海道 品川インターシティー店)
> >>
> >> Location:
> >>
> >> https://www.google.com/maps/d/edit?mid=zBFFkY6dvVno.kOTkyNjZ2oU0=sharing
> >> 5th floor at the "shop and restaurant building" (between A and B
> >> buildings).
> >> It is at the opposite side of JR Shinagawa Station from the Summit side.)

1. Go to Shinagawa station.
2. Climb the stairs (or take an escalator) and go through the station
   to the opposite side (Ko-nan exit).
3. You are on a pedestrian deck.  Follow the red-dotted route in the
   following map.  The restrant is in the "Shop & Restaurant" bldg.
   http://www.sicity-sr.com/access.html

My feeling is the route is a bit tricky and it's best to catch some
locals at the venue and follow her/him.  It should be 20~30 minutes walk.

> >> Approximately it costs ~5000 (Japanese) Yen depending on the number of
> >> folks who join.
> >> Note that we have no sponsors.
> >>
> >> If you have any trouble in reaching there or some question, reach me
> >> @ritchey98 on Twitter.
> >>
> >> 
> >>
> >> See you in Tokyo!
> >> Akihiro
> >>
> >> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> > --
> > El Psy Congroo,
> > Antonio Añon Brasolin,
> > Masters Degree Student in Universidade Federal de São Carlos
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] OpenStack Meiji (明治) - Our next release name has been selected

[IWAMOTO Toshihiro]

It is totally understandable that the combination of Japan and
Meiji recalls harsh history to some people, especially in East Asia.
As a Japanese, I'm sorry for that.

I think the release naming process should not cause such friction.
Could we just select some other neutral candidate name instead?

At Wed, 08 Jul 2015 04:40:58 +,
Jaesuk Ahn wrote:
 
 Dear OpenStack Community,
 
 As Clint mentioned, there is some historical background regarding the name
 of Meiji.
 
 I have been aware of the potential problem with Meiji, however, I have
 not raise my voice here since it was not the top of the list.
 However, with current situation, IMHO, it is better to present my concern
 over this name.
 
 It was briefly mentioned in the previous email thread.
 Once again, please understand that the name Meiji can create some
 historical debate in East Asia, especially in Korea. Under the name of
 Emperor Meiji, Japan forcefully colonized Korea. Lots of people in Korea
 suffered during the colonization. Furthermore, this history is not the one
 only exists on the book. This is actually very recent event in terms of
 history.
 
 FYI, there is a wiki entry:
 https://en.wikipedia.org/wiki/Korea_under_Japanese_rule
 If you look at legacy section in the above wiki entry, you will
 understand what I am referring to.
 
 I am not saying the word Meiji is bad in general. I just want to state
 that this word can cause very negative feeling to some people in East Asia.
 
 I totally value the community's process and rules. I recognize that process
 to select Meiji itself was not the problem at all.
 However, I am sincerely asking the community to acknowledge my concern over
 this name, and please put some effort as a community to avoid any
 unnecessary debate.
 
 Thank you.
 
 Best wishes for everyone here in the community.
 
 
 Jaesuk Ahn, Ph.D.
 member of OpenStack Community and OpenStack Korea User Group.
 
 
 
 
 On Wed, Jul 8, 2015 at 2:24 AM Clint Byrum cl...@fewbar.com wrote:
 
  http://afe.easia.columbia.edu/main_pop/kpct/kp_meiji.htm
 
  There's a summary of what Meiji might mean to historians. I'm not sure if
  OpenStack needs to be too concerned about this, but I believe this is what
  Ian is referring to as historical and political debates in East Asia.
 
  Seems like a hot-button name for OpenStack to adopt.
 
  Excerpts from Ian Y. Choi's message of 2015-07-07 05:40:52 -0700:
   Hello,
  
   I think the third name 'Meiji' may arise some historical and political
   debates in East Asia, as far as I know.
  
   Could you share what significant identified problems are on the first two
   selected from our election?
  
  
   With many thanks,
  
   /Ian
  
   On Tue, Jul 7, 2015 at 8:55 PM, Masayuki Igawa masayuki.ig...@gmail.com
  
   wrote:
  
Hi,
# I'm a native Japanese speaker :-)
   
2015年7月7日(火) 20:33 Amrith Kumar amr...@tesora.com:
   
Maybe this (the google answer)?
   
www.youtube.com/watch?v=Qvw0A12aOGQ
   
   
Yeah, this is correct pronunciation.
   
   
But someone who is a native Japanese speaker should confirm.
   
   
https://www.youtube.com/watch?v=D9vwge557hY
I think this is a casual version and Japanese people are familiar with
this.
   
Thanks,
-- Masayuki Igawa
   
   
-amrith
   
P.S. the yahoo answer suggests that you pronounce it as Meh - gee
  with
the emphasis on the meh ;)
   
-Original Message-
From: Matthias Runge [mailto:mru...@redhat.com]
Sent: Tuesday, July 07, 2015 6:08 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] OpenStack Meiji (明治) - Our next release
  name
has been selected
   
On Mon, Jul 06, 2015 at 11:48:13PM -0400, Monty Taylor wrote:
 significant identified problems... but the third in the list, Meiji
  (明
 治) is clear.

 So please join me in welcoming the latest name to our family ... and
 if you, like me, are not a native Japanese speaker, in learning two
 new characters.
   
Could someone provide a hint please, how to pronounce this properly?
--
Matthias Runge mru...@redhat.com
   
   
  __
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
   
  __
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
   
--
-- Masayuki Igawa
   
   
  __
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
  

Re: [openstack-dev] [Neutron] LeastNetwork scheduling for DHCP

[IWAMOTO Toshihiro]

At Thu, 6 Nov 2014 15:39:25 +,
Narasimhan, Vivekanandan wrote:
 
 Hi Neutron Stackers,
 
 
 
 There is an interest among vendors to bring Least Networks scheduling for 
 DHCP into Openstack Neutron.
 
 
 
 Currently there are the following blueprints lying there, all of them trying 
 to address this issue:
 
 https://review.openstack.org/111210
 
 https://review.openstack.org/#/c/130912/
 
 https://review.openstack.org/104587
 

 We are trying  to pull together all these BPs as one Umbrella BP, on which we
 can pour volunteers from every side, to clear out this BP itself as initial 
 step.

 So we would like to collaborate, to plan BP approval for these.

 Please respond if you are interested.

Oui/Yes.

But, OTOH, this doesn't need to be very complex.
I have code that does the job based on the L3 HA scheduler code at
home.

--
IWAMOTO Toshihiro

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] Updated Object Model?

[IWAMOTO Toshihiro]

It is pity to see enoumous LBaaS efforts have been largely spinning
(in a sense of spinlocks) for a while.

At Thu, 15 May 2014 14:31:54 +0400,
Eugene Nikanorov wrote:
 
 [1  multipart/alternative (7bit)]
 [1.1  text/plain; UTF-8 (7bit)]
 Hi Craig,
 
 Implementation-specific options are not exposed through the API, or
 otherwise it would be inconsistent, given that we are moving to a
 flavor-based approach of specifying service requirements where
 implementation is completely hidden from the user.

There was a lot of arguments against your flavor proposal at the
design summit session and at the meeting at the pod area.
So, it is not clear if moving to a flavor-based happens in a
reasonalbe timeframe.

OTOH, the flavor framework is not much more than a bitmap matching of
feature vectors.  I think something is not going right as we spent
good 30mins on this topic at the pod area.

We'll be able to continue the same technical level argument at home as
we did for the last couple of months.

My suggestion is to try to discuss differently here at the summit.

--
IWAMOTO Toshihiro

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][NFV] NFV BoF at design summit

[IWAMOTO Toshihiro]

At Wed, 14 May 2014 14:40:03 -0700,
punal patel wrote:
 
 [1  multipart/alternative (7bit)]
 [1.1  text/plain; UTF-8 (7bit)]
 Will this be recorded? or can I join webex?
 

There's no official recording facility.  You may be able to ask
someone to record or stream.
BTW, I think it is a good idea to try to take discussion memo on
etherpad, just as official design summit programs.


 On Wed, May 14, 2014 at 10:06 AM, Chris Wright chr...@redhat.com wrote:
 
  Thursday at 1:30 PM in the Neutron Pod we'll do
  an NFV BoF.  If you are at design summit and
  interested in Neutron + NFV please come join us.
 
  thanks,
  -chris
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 [1.2  text/html; UTF-8 (quoted-printable)]
 
 [2  text/plain; us-ascii (7bit)]
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] Updated Object Model?

[IWAMOTO Toshihiro]

At Wed, 14 May 2014 10:18:29 -0700,
Stephen Balukoff wrote:
 
 [1  multipart/alternative (7bit)]
 [1.1  text/plain; UTF-8 (7bit)]
 Hi Craig,
 
 I'm attaching the latest object model diagram as discussed with the RAX
 team last night, Samuel and Eugene. Note that we don't necessarily have
 HP's blessing on this model yet, nor do we have Neutron core developer buy
 in.  But in any case, here it is.

Sorry for jumping into a meta-argument,
but what the LBaaS community needs to do first is to figure out how to
make the neutron community agree on a LBaaS proposal.
On the other hand, the neutron community (or the core?) needs to make
it clear that what criteria or process is required to approve some
LBaaS idea (obj. model, API, whatsoever).


I'm sorry to say (again) that not much people have energy to follow
and check out small differences in those large pictures of LBaaS
object models.

--
IWAMOTO Toshihiro

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][QoS] Interest in a meeting at the Networking pod at the design summit?

[IWAMOTO Toshihiro]

At Tue, 6 May 2014 16:19:22 +,
Collins, Sean wrote:
 
 For those attending, to discuss the QoS API current status?
 

I'd like to attend (as a proxy for Itsuro Oda), too.

I'm wondering if the Neutron flavor framework could be used/enhanced
for QoS purposes.
The framework currently tries to deal with qualitative attributes
(eg. existence of specific features), QoS deals with quantitative
attributes (eg. bandwidth etc.).  IMO, handling of them aren't
entirely different.

--
IWAMOTO Toshihiro

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] virtual resource for service chaining

[IWAMOTO Toshihiro]

Hi Neutron advanced service folks,

As you may have noticed, there is a session slot for virtual resource
for service chaining.  Please check the following etherpad.

https://etherpad.openstack.org/p/juno-virtual-resource-for-service-chaining

I'd like to have some preliminary discussion on the etherpad or on
the mailing list, as the session slot is quite limited.

Thanks.

--
IWAMOTO Toshihiro

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][L3] Agent manager customization

[IWAMOTO Toshihiro]

At Thu, 24 Apr 2014 15:24:53 +0900,
IWAMOTO Toshihiro wrote:
 
 At Thu, 24 Apr 2014 00:34:36 +0200,
 ZZelle wrote:
  
  [1  multipart/alternative (7bit)]
  [1.1  text/plain; ISO-8859-1 (7bit)]
  Hi Carl,
  
  
   A clear l3 agent manager interface with hookable methods would clearly
  simplify the understanding, the dev/support.
 
 A reasonable set of hook points in the l3-agent would be great for
 decoupling FWaaS and VPNaaS codes from the l3-agent, too.

The following spec for l3-agent-consolidation BP might be of your
interest.

https://review.openstack.org/#/c/91532/

--
IWAMOTO Toshihiro

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][L3] Agent manager customization

[IWAMOTO Toshihiro]

At Thu, 24 Apr 2014 00:34:36 +0200,
ZZelle wrote:
 
 [1  multipart/alternative (7bit)]
 [1.1  text/plain; ISO-8859-1 (7bit)]
 Hi Carl,
 
 
  A clear l3 agent manager interface with hookable methods would clearly
 simplify the understanding, the dev/support.

A reasonable set of hook points in the l3-agent would be great for
decoupling FWaaS and VPNaaS codes from the l3-agent, too.

--
IWAMOTO Toshihiro

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

[IWAMOTO Toshihiro]

At Thu, 13 Mar 2014 07:48:53 -0700,
Aaron Rosen wrote:
 
 [1  multipart/alternative (7bit)]
 [1.1  text/plain; ISO-8859-1 (7bit)]
 The easiest/quickest thing to do for ice house would probably be to run the
 initial sync in parallel like the dhcp-agent does for this exact reason.
 See: https://review.openstack.org/#/c/28914/ which did this for thr
 dhcp-agent.
 
 Best,
 
 Aaron
 On Thu, Mar 13, 2014 at 12:18 PM, Miguel Angel Ajo majop...@redhat.comwrote:
 
  Yuri, could you elaborate your idea in detail? , I'm lost at some
  points with your unix domain / token authentication.
 
  Where does the token come from?,
 
  Who starts rootwrap the first time?
 
  If you could write a full interaction sequence, on the etherpad, from
  rootwrap daemon start ,to a simple call to system happening, I think that'd
  help my understanding.
 
 
 Here it is: https://etherpad.openstack.org/p/rootwrap-agent
 Please take a look.

I've added a couple of security-related comments (pickle decoding and
token leak) on the etherpad.
Please check.

--
IWAMOTO Toshihiro


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][rootwrap] Performance considerations, sudo?

[IWAMOTO Toshihiro]

At Wed, 05 Mar 2014 15:42:54 +0100,
Miguel Angel Ajo wrote:
 3) I also find 10 minutes a long time to setup 192 networks/basic tenant 
 structures, I wonder if that time could be reduced by conversion
 of system process calls into system library calls (I know we don't have
 libraries for iproute, iptables?, and many other things... but it's a
 problem that's probably worth looking at.)

Try benchmarking

   $ sudo ip netns exec qfoobar /bin/echo

Network namespace switching costs almost as much as a rootwrap
execution, IIRC.

Execution coalesceing is not enough in this case and we would need to
change how Neutron issues commands, IMO.




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] Object Model discussion

[IWAMOTO Toshihiro]

At Thu, 20 Feb 2014 15:21:49 +0400,
Eugene Nikanorov wrote:
 
 Hi Iwamoto,
 
 
  I agree with Samuel here.  I feel the logical model and other issues
  (implementation etc.) are mixed in the discussion.
 
 
 A little bit. While ideally it's better to separate it, in my opinion we
 need to have some 'fair bit' of implementation details
 in API in order to reduce code complexity (I'll try to explain it on the
 meeting). Currently these 'implementation details' are implied because we
 deal with simplest configurations which maps 1:1 to a backend.

Exposing some implementation details as API might not be ideal but
would be accepable if it saves a lot of code complexity.

  I'm failing to understand why the current model is unfit for L7 rules.
 
- pools belonging to a L7 group should be created with the same
  provider/flavor by a user
- pool scheduling can be delayed until it is bound to a vip to make
  sure pools belonging to a L7 group are scheduled to one backend
 
  While that could be an option, It's not as easy as it seems.
 We've discussed that back on HK summit but at that point decided that it's
 undesirable.

Could you give some more details/examples why my proposal above is
undesirable?
In my opinion, pool rescheduling happens anyway when an agent dies,
and calculating pool-vip grouping based on their connectivity is not hard.


   I think grouping vips and pools is important part of logical model, even
  if
   some users may not care about it.
 
  One possibility is to provide an optional data structure to describe
  grouping of vips and pools, on top of the existing pool-vip model.
 
 That would be 'loadbalancer' approach, #2 in a wiki page.
 So far we tend to introduce such grouping directly into vip-pool
 relationship.
 I plan to explain that in more detail on the meeting.

My idea was to keep the 'loadbalancer' API optional for users who
don't care about grouping.

--
IWAMOTO Toshihiro


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] Object Model discussion

[IWAMOTO Toshihiro]

At Tue, 18 Feb 2014 18:47:37 -0800,
Stephen Balukoff wrote:
 
 [1  multipart/alternative (7bit)]
 [1.1  text/plain; UTF-8 (7bit)]
 Small correction to my option #4 (here as #4.1). Neutron port_id should be
 an attribute of the 'loadbalancer' object, not the 'cluster' object.
 (Though cluster should have a network_id attribute).

Hi Eugene and Stephen,
I'd like to see the wiki updated with the plan #4 and current issues
as mentioned in emails.  It'll greatly help me to keep in touch with
the discussion.

Thanks.
--
IWAMOTO Toshihiro

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][Neutron] About paramiko's SSHException: Error reading SSH protocol banner

[IWAMOTO Toshihiro]

At Fri, 27 Dec 2013 01:53:59 +0100,
Salvatore Orlando wrote:
 
 [1  multipart/alternative (7bit)]
 [1.1  text/plain; ISO-8859-1 (7bit)]
 I put together all the patches which we prepared for making parallel
 testing work, and ran a few times 'check experimental' on the gate to see
 whether it worked or not.
 
 With parallel testing, the only really troubling issue are the scenario
 tests which require to access a VM from a floating IP, and the new patches
 we've squashed together in [1] should address this issue. However, the
 result is that timeouts are still observed but with a different message [2].
 I'm not really familiar with it, and I've never observed it in local
 testing. I wonder if it just happens to be the usual problem about the
 target host not being reachable, or if it is something specific to paramiko.
 
 Any hint would be appreciated, since from the logs is appears everything is
 wired properly.

It seems that a TCP connection has been established but paramiko
failed get data from the server in time.  Does increasing paramiko
timeout help?

 [1] https://review.openstack.org/#/c/57420/
 [2]
 http://logs.openstack.org/20/57420/40/experimental/check-tempest-dsvm-neutron-isolated-parallel/a74bdc8/console.html#_2013-12-26_22_51_31_817
 [1.2  text/html; ISO-8859-1 (quoted-printable)]
 
 [2  text/plain; us-ascii (7bit)]
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] Thursday subteam meeting

[IWAMOTO Toshihiro]

At Fri, 29 Nov 2013 07:25:54 +0900,
Itsuro ODA wrote:
 
 Hi Eugene,
 
 Thank you for the response.
 
 I have a comment.
 I think 'provider' attribute should be added to loadbalance resource
 and used rather than pool's 'provider' since I think using multiple
 driver within a loadbalancer does not make sense.

There can be a 'provider' attribute in a loadbalancer resource, but,
to maintain API, the 'provider' attribute in pools should remain the
same.
Is there any other attribute planned for the loadbalancer resource?

 What do you think ?
 
 I'm looking forward to your code up !
 
 Thanks.
 Itsuro Oda
 
 On Thu, 28 Nov 2013 16:58:40 +0400
 Eugene Nikanorov enikano...@mirantis.com wrote:
 
  Hi Itsuro,
  
  I've updated the wiki with some examples of cli workflow that illustrate
  proposed API.
  Please see the updated page:
  https://wiki.openstack.org/wiki/Neutron/LBaaS/LoadbalancerInstance#API_change
  
  Thanks,
  Eugene.

--
IWAMOTO Toshihiro

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] Loadbalancer instance design.

[IWAMOTO Toshihiro]

At Fri, 15 Nov 2013 17:14:47 +0400,
Eugene Nikanorov wrote:
 Hi folks,
 
 I've created a brief description of this feature.
 You can find it here:
 https://wiki.openstack.org/wiki/Neutron/LBaaS/LoadbalancerInstancehttps://blueprints.launchpad.net/neutron/+spec/lbaas-service-instance
 
 I would appreciate any comments/ideas about this.

How do you plan to handle API compatibility?
I think that was a major part of the discussion at the design summit.

--
IWAMOTO Toshihiro

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] L3 router service integration with Service Type Framework

[IWAMOTO Toshihiro]

Let me try to clarify things a bit.

At Fri, 25 Oct 2013 08:44:05 +0900,
Itsuro ODA wrote:
 
 Hi Gray,
 
 Thanks for your response.
 
 Our plan is as follows:
 * LVS driver is one of lbaas provider driver.
   It communicates with l3_agent instead of lbaas_agent.

LVS agent part will reside in the L3 agent, just like FWaaS.
At this point, I plan to use the same LBaaS RPC topic, so the server
(Neutron) side of the LVS implementation isn't much different from
the current HAproxy one.

 * For l3_agent side, I think the implementation is same as
   fwaas. L3NATAgent inherits like LVSL3AgentRpcCallback
   class added which communicates with LVS provider driver.
   So l3_agent functions already existed are not changed,
   just added LB function.

L3NATAgent inherits FWaaSL3AgentRpcCallback.
My plan is to create LVSL3AgentRpcCallback and make L3NATAgent inherit
this. LVSL3AgentRpcCallback will make use of the L3 router ports for
LVS operations.

   I think the implementation would change depending on
   the service chaining discussion.

Exactly.
There is a private LVS LBaaS implementation.
I think the above plan is a good way to provide a working example
based on the code.
Implementations can change as the service chaining discussion
develops.


 Thanks,
 Itsuto Oda
 
 # note that Toshihiro Iwamoto, a main developer of our BPs
 # may replay instead of me. He will attend the HK summit.
 
 On Thu, 24 Oct 2013 16:03:25 -0700
 Gary Duan gd...@varmour.com wrote:
 
  Hi, Oda-san,
  
  Thanks for your response.
  
  L3 agent function should remain the same, as one driver implementation of
  L3 router plugin.
  
  My understanding is your lbaas driver can be running on top of L3 agent and
  LVS' own routing services. Is my understanding correct?
  
  Thanks,
  Gary
  
  
  On Thu, Oct 24, 2013 at 3:16 PM, Itsuro ODA o...@valinux.co.jp wrote:
  
   Hi,
  
   We are going to implement 2-arm type lbaas using LVS, and have submitted
   the following BPs.
  
  
   https://blueprints.launchpad.net/neutron/+spec/lbaas-support-routed-service-insertion
   https://blueprints.launchpad.net/neutron/+spec/lbaas-lvs-driver
   https://blueprints.launchpad.net/neutron/+spec/lbaas-lvs-extra-features
  
   Maybe the first one is same as yours.
   We are happy if we just concentrate making a provider driver.
  
   Thanks.
   Itsuro Oda
  
   On Thu, 24 Oct 2013 11:56:53 -0700
   Gary Duan gd...@varmour.com wrote:
  
Hi,
   
I've registered a BP for L3 router service integration with service
framework.
   
https://blueprints.launchpad.net/neutron/+spec/l3-router-service-type
   
In general, the implementation will align with how LBaaS is integrated
   with
the framework. One consideration we heard from several team members is 
to
be able to support vendor specific features and extensions in the 
service
plugin.
   
Any comment is welcome.
   
Thanks,
Gary
  
   --
   Itsuro ODA o...@valinux.co.jp
  
  
   ___
   OpenStack-dev mailing list
   OpenStack-dev@lists.openstack.org
   http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
  
 
 -- 
 Itsuro ODA o...@valinux.co.jp
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev