Re: [openstack-dev] [Neutron] Neutron team social event in Barcelona

[Ben Pfaff]

On Fri, Oct 14, 2016 at 01:30:57PM -0500, Miguel Lavalle wrote:
> It is easy to get there by subway from the Summit venue:
> https://goo.gl/maps/HjaTEcBbDUR2. I made a reservation for 25 people under
> 'Neutron' or "Miguel Lavalle". Please confirm your attendance so we can get
> a final count.

I'm planning to attend.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ovs-dev] The QoS feature minimum guaranteed bandwidth of OpenvSwitch not work

[Ben Pfaff]

On Sat, Jul 02, 2016 at 03:17:21PM +, Xiao Ma (xima2) wrote:
> Hi , Ben
> 
> Thanks for your reply.
> 
> I configured the bond0 using linux bond then the result is not good for 
> minimum guaranteed bandwidth.
> But if I use ovs-vsctl add-bond to configure the bond0 interface, the result 
> is good.
> If I use linux bond with linux tc, the tc works well also.
> 
>   However, most problems with QoS on Linux are not bugs in Open
>   vSwitch at all.  They tend to be either configuration errors
>   (please see the earlier questions in this section) or issues with
> 
> Could you send  me the section link?

https://github.com/openvswitch/ovs/blob/master/FAQ.md#quality-of-service-qos

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ovs-dev] The QoS feature minimum guaranteed bandwidth of OpenvSwitch not work

[Ben Pfaff]

On Fri, Jul 01, 2016 at 03:40:30AM +, Xiao Ma (xima2) wrote:
> I want to use the QoS feature of OpenvSwitch to control the bandwidth based 
> on the vlan id(Scene 1) or port id(Scene 2).
> So I deployed it as showed bellow,and configured the qos rules,the flows,and 
> used iperf tool to test it.
> But the result is disappointment.

Are you confident that this is actually an OVS problem and not a problem
with the Linux kernel HTB code?  The FAQ says:

### Q: I configured QoS, correctly, but my measurements show that it isn't
   working as well as I expect.

A: With the Linux kernel, the Open vSwitch implementation of QoS has
   two aspects:

   - Open vSwitch configures a subset of Linux kernel QoS
 features, according to what is in OVSDB.  It is possible that
 this code has bugs.  If you believe that this is so, then you
 can configure the Linux traffic control (QoS) stack directly
 with the "tc" program.  If you get better results that way,
 you can send a detailed bug report to b...@openvswitch.org.

 It is certain that Open vSwitch cannot configure every Linux
 kernel QoS feature.  If you need some feature that OVS cannot
 configure, then you can also use "tc" directly (or add that
 feature to OVS).

   - The Open vSwitch implementation of OpenFlow allows flows to
 be directed to particular queues.  This is pretty simple and
 unlikely to have serious bugs at this point.

   However, most problems with QoS on Linux are not bugs in Open
   vSwitch at all.  They tend to be either configuration errors
   (please see the earlier questions in this section) or issues with
   the traffic control (QoS) stack in Linux.  The Open vSwitch
   developers are not experts on Linux traffic control.  We suggest
   that, if you believe you are encountering a problem with Linux
   traffic control, that you consult the tc manpages (e.g. tc(8),
   tc-htb(8), tc-hfsc(8)), web resources (e.g. http://lartc.org/), or
   mailing lists (e.g. http://vger.kernel.org/vger-lists.html#netdev).

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][networking-ovn] OVN vs. OpenDayLight

[Ben Pfaff]

On Thu, Jun 09, 2016 at 10:28:31AM -0700, rezroo wrote:
> I'm trying to reconcile differences and similarities between OVN and
> OpenDayLight in my head. Can someone help me compare these two technologies
> and explain if they solve the same problem, or if there are fundamental
> differences between them?

OVN implements network virtualization for clouds of VMs or containers or
a mix.  Open Daylight is a platform for managing networks that can do
anything you want.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] support of NSH in networking-SFC

[Ben Pfaff]

I'm probably the wrong person to give advice on kernel development,
since I haven't been involved in it for years.  I just know that it's
difficult, and not always because of the code.

It's hard to support a protocol in OVS before it's supported in the
kernel, since userspace without a kernel implementation is not very
useful.

On Wed, Jun 01, 2016 at 09:59:12PM +, Elzur, Uri wrote:
> Hi Ben
> 
> Any guidance you can offer will be appreciated. The process has taken long 
> time and precious cycles. How can we get to a coordinated Kernel and OvS 
> approach to avoid the challenges and potentially misaligned advise we got 
> (per Yi Yang's mail)?
> 
> Thx
> 
> Uri ("Oo-Ree")
> C: 949-378-7568
> 
> 
> -Original Message-
> From: Ben Pfaff [mailto:b...@ovn.org] 
> Sent: Tuesday, May 31, 2016 9:48 PM
> To: OpenStack Development Mailing List (not for usage questions) 
> <openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [Neutron] support of NSH in networking-SFC
> 
> On Wed, Jun 01, 2016 at 12:08:23AM +, Yang, Yi Y wrote:
> > Ben, yes, we submitted nsh support patch set last year, but ovs 
> > community told me we have to push kernel part into Linux kernel tree, 
> > we're struggling to do this, but something blocked us from doing this.
> 
> It's quite difficult to get patches for a new protocol into the kernel.
> You have my sympathy.
> 
> > Recently, ovs did some changes in tunnel protocols which requires the 
> > packet decapsulated by a tunnel must be a Ethernet packet, but Linux 
> > kernel (net-next) tree accepted VxLAN-gpe patch set from Redhat guy 
> > (Jiri Benc) which requires the packet decapsulated by VxLAN-gpe port 
> > must be L3 packet but not L2 Ethernet packet, this blocked us from 
> > progressing better.
> > 
> > Simon Horman (Netronome guy) has posted a series of patches to remove 
> > the mandatory requirement from ovs in order that the packet from a 
> > tunnel can be any packet, but so far we didn't see they are merged.
> 
> These are slowly working their way through OVS review, but these also have a 
> prerequisite on kernel patches, so it's not easy to get them in either.
> 
> > I heard ovs community looks forward to getting nsh patches merged, it 
> > will be great if ovs guys can help progress this.
> 
> I do plan to do my part in review (but much of this is kernel review, which 
> I'm not really involved in anymore).
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] support of NSH in networking-SFC

[Ben Pfaff]

On Wed, Jun 01, 2016 at 12:08:23AM +, Yang, Yi Y wrote:
> Ben, yes, we submitted nsh support patch set last year, but ovs
> community told me we have to push kernel part into Linux kernel tree,
> we're struggling to do this, but something blocked us from doing this.

It's quite difficult to get patches for a new protocol into the kernel.
You have my sympathy.

> Recently, ovs did some changes in tunnel protocols which requires the
> packet decapsulated by a tunnel must be a Ethernet packet, but Linux
> kernel (net-next) tree accepted VxLAN-gpe patch set from Redhat guy
> (Jiri Benc) which requires the packet decapsulated by VxLAN-gpe port
> must be L3 packet but not L2 Ethernet packet, this blocked us from
> progressing better.
> 
> Simon Horman (Netronome guy) has posted a series of patches to remove
> the mandatory requirement from ovs in order that the packet from a
> tunnel can be any packet, but so far we didn't see they are merged.

These are slowly working their way through OVS review, but these also
have a prerequisite on kernel patches, so it's not easy to get them in
either.

> I heard ovs community looks forward to getting nsh patches merged, it
> will be great if ovs guys can help progress this.

I do plan to do my part in review (but much of this is kernel review,
which I'm not really involved in anymore).

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] support of NSH in networking-SFC

[Ben Pfaff]

On Mon, May 30, 2016 at 10:12:34PM -0400, Paul Carver wrote:
> I don't know the details of why OvS hasn't added NSH support so I can't
> judge the validity of the concerns, but one way or another there has to be a
> production-quality dataplane for networking-sfc to front-end.

It looks like the last time anyone submitted NSH patches to Open vSwitch
was September 2015.  They got some reviews but no new version has been
posted since.

Basically, we can't add NSH support if no one submits patches.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OVN] [networking-ovn] [networking-sfc] SFC and OVN

[Ben Pfaff]

On Wed, May 25, 2016 at 09:27:31AM -0500, Ryan Moats wrote:
> As I understand it, Table 0 identifies the logical port and logical
> flow. I'm worried that this means we'll end up with separate bucket
> rules for each ingress port of the port pairs that make up a port
> group, leading to a cardinality product in the number of rules.
> I'm trying to think of a way where Table 0 could identify the packet
> as being part of a particular port group, and then I'd only need one
> set of bucket rules to figure out the egress side.  However, the
> amount of free metadata space is limited and so before we go down
> this path, I'm going to pull Justin, Ben and Russell in to see if
> they buy into this idea or if they can think of an alternative.

I've barely been following the discussion, so a recap of the question
here would help a lot.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Social at the summit

[Ben Pfaff]

Sign me up, too.

On Mon, Apr 25, 2016 at 01:07:02PM -0500, Kyle Mestery wrote:
> OK, there is enough interest, I'll find a place on 6th Street for us
> and get a reservation for Thursday around 7 or so.
> 
> Thanks folks!
> 
> On Mon, Apr 25, 2016 at 12:30 PM, Zhou, Han  wrote:
> > +1 :)
> >
> > Han Zhou
> > Irc: zhouhan
> >
> >
> > On Monday, April 25, 2016, Korzeniewski, Artur
> >  wrote:
> >>
> >> Sign me up :)
> >>
> >> Artur
> >> IRC: korzen
> >>
> >> -Original Message-
> >> From: Darek Smigiel [mailto:smigiel.dari...@gmail.com]
> >> Sent: Monday, April 25, 2016 7:19 PM
> >> To: OpenStack Development Mailing List (not for usage questions)
> >> 
> >> Subject: Re: [openstack-dev] [neutron] Social at the summit
> >>
> >> Count me in!
> >> Will be good to meet all you guys!
> >>
> >> Darek (dasm) Smigiel
> >>
> >> > On Apr 25, 2016, at 12:13 PM, Doug Wiegley
> >> >  wrote:
> >> >
> >> >
> >> >> On Apr 25, 2016, at 12:01 PM, Ihar Hrachyshka 
> >> >> wrote:
> >> >>
> >> >> WAT???
> >> >>
> >> >> It was never supposed to be core only. Everyone is welcome!
> >> >
> >> > +2
> >> >
> >> > irony intended.
> >> >
> >> > Socials are not controlled by gerrit ACLs.  :-)
> >> >
> >> > doug
> >> >
> >> >>
> >> >> Sent from my iPhone
> >> >>
> >> >>> On 25 Apr 2016, at 11:56, Edgar Magana 
> >> >>> wrote:
> >> >>>
> >> >>> Would you extend it to ex-cores?
> >> >>>
> >> >>> Edgar
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >>  On 4/25/16, 10:55 AM, "Kyle Mestery"  wrote:
> >> 
> >>  Ihar, Henry and I were talking and we thought Thursday night makes
> >>  sense for a Neutron social in Austin. If others agree, reply on this 
> >>  thread
> >>  and we'll find a place.
> >> 
> >>  Thanks!
> >>  Kyle
> >> 
> >>  ___
> >>  ___ OpenStack Development Mailing List (not for usage
> >>  questions)
> >>  Unsubscribe:
> >>  openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >>  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>> 
> >> >>> __ OpenStack Development Mailing List (not for usage questions)
> >> >>> Unsubscribe:
> >> >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>
> >> >> _
> >> >> _ OpenStack Development Mailing List (not for usage questions)
> >> >> Unsubscribe:
> >> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >> >
> >> > __
> >> >  OpenStack Development Mailing List (not for usage questions)
> >> > Unsubscribe:
> >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Testing Neutron with latest OVS

[Ben Pfaff]

On Thu, Jan 14, 2016 at 09:30:03PM +, Sean M. Collins wrote:
> On Thu, Jan 14, 2016 at 03:13:16PM CST, Clark Boylan wrote:
> > Forgive my ignorance, but if we are gating on OVS 2.0 that is because it
> > is what is shipped with the distros that we test on. Are we saying that
> > no one uses the distro provided OVS packages to run Neutron? If not what
> > are they using?
> 
> Right - this was my impression as well.

Even Debian stable (jessie) has OVS 2.3, what distro has 2.0?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Testing Neutron with latest OVS

[Ben Pfaff]

On Thu, Jan 14, 2016 at 01:45:39PM -0800, Clark Boylan wrote:
> On Thu, Jan 14, 2016, at 01:32 PM, Ben Pfaff wrote:
> > On Thu, Jan 14, 2016 at 09:30:03PM +, Sean M. Collins wrote:
> > > On Thu, Jan 14, 2016 at 03:13:16PM CST, Clark Boylan wrote:
> > > > Forgive my ignorance, but if we are gating on OVS 2.0 that is because it
> > > > is what is shipped with the distros that we test on. Are we saying that
> > > > no one uses the distro provided OVS packages to run Neutron? If not what
> > > > are they using?
> > > 
> > > Right - this was my impression as well.
> > 
> > Even Debian stable (jessie) has OVS 2.3, what distro has 2.0?
> > 
> For the three distro releases we currently test on
> Ubuntu Trusty: 2.0.2
> CentOS7: 2.4.0
> Fedora 23 (not actually used just yet, but is being worked on): 2.4.0

OK, I misunderstood the problem.  It's not "distros have old OVS" but
"we're using an old distro with old OVS".  Fair enough.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Sender Auth Failure] Re: [neutron] How could an L2 agent extension access agent methods ?

[Ben Pfaff]

On Tue, Dec 15, 2015 at 05:58:14PM +, Frances, Margaret wrote:
> 2. OpenFlow¹s Goto instruction directs a frame from one table to the next.
>  A redirection in this sense must be to a higher-numbered table, which is
> to say that OF pipeline processing can only go forward (see p.18, para.2
> of the 1.4.1 spec 
>  specifications/openflow/openflow-switch-v1.4.1.pdf>).  However, OvS (at
> least v2.0.2) implements a resubmit action, which re-searches another
> table‹higher-, lower-, or even same-numbered‹and executes any actions
> found there in addition to any subsequent actions in the current flow
> entry.  It is by using resubmit that the proposed design could work, as
> shown in the ovs-ofctl command posted here
> 
> .  (Maybe there are other ways, too.)  The resubmit action is a Nicira
> vendor extension that at least at one point, and maybe still, was known to
> be implemented only by OvS.  I mention this because I wonder if the
> proposed design (and my sample command) calls for flow traversal in a
> manner not explicitly supported by OpenFlow and so may not work in future
> versions of OvS.

OVS has has "resubmit" for a long time and it's heavily used by lots of
projects.  It's not going to go away.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][networking-ovn][l3] Add/delete router interface

[Ben Pfaff]

On Fri, Oct 16, 2015 at 07:05:44PM -0700, Chandra Sekhar Vejendla wrote:
> The reason for the failure of these tests has got to do with the way
> OVN handles router interface. In openstack multiple subnets can be
> added to a network and for each of these subnets the router interface
> has to be explicitly configured. So for a given network there can be
> multiple router interfaces. With the current NB schema in OVN there
> can be only 1 router interface for a network.

I've just posted patches that support multiple routed subnets for a
logical switch, starting here:
http://openvswitch.org/pipermail/dev/2015-October/061356.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] New cycle started. What are you up to, folks?

[Ben Pfaff]

On Fri, Oct 02, 2015 at 08:19:47AM +0300, Gal Sagie wrote:
> *OVN*
> 
>1) OVN integration with Kuryr

Can you say anything more about what that entails?

Thanks,

Ben.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] New cycle started. What are you up to, folks?

[Ben Pfaff]

On Fri, Oct 02, 2015 at 10:07:32PM +0300, Gal Sagie wrote:
> If you are not familiar with Kuryr you can read my blog post about it here
> [1].
> Basically we already have a working demo integrating with Neutron and we
> are going to show
> it in OpenStack Tokyo (probably in the keynotes and in a specific Kuryr
> session).
> 
> For the OVN part, there are some areas that i want to work on:

Since it sounds like you're targeting a lot for Tokyo later this month,
I guess I can learn more then.  Thanks!

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] PTL Non-Candidacy

[Ben Pfaff]

Are you planning to remain involved with OpenStack?

On Fri, Sep 11, 2015 at 2:12 PM, Kyle Mestery  wrote:
> I'm writing to let everyone know that I do not plan to run for Neutron PTL
> for a fourth cycle. Being a PTL is a rewarding but difficult job, as Morgan
> recently put it in his non-candidacy email [1]. But it goes further than
> that for me. As Flavio put it in his post about "Being a PTL" [2], it's a
> full time job. In the case of Neutron, it's more than a full time job, it's
> literally an always on job.
>
> I've tried really hard over my three cycles as PTL to build a stronger web
> of trust so the project can grow, and I feel that's been accomplished. We
> have a strong bench of future PTLs and leaders ready to go, I'm excited to
> watch them lead and help them in anyway I can.
>
> As was said by Zane in a recent email [3], while Heat may have pioneered the
> concept of rotating PTL duties with each cycle, I'd like to highly encourage
> Neutron and other projects to do the same. Having a deep bench of leaders
> supporting each other is important for the future of all projects.
>
> See you all in Tokyo!
> Kyle
>
> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2015-September/074157.html
> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2015-September/073986.html
> [2]
> http://lists.openstack.org/pipermail/openstack-dev/2015-September/074242.html
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
"I don't normally do acked-by's.  I think it's my way of avoiding
getting blamed when it all blows up."   Andrew Morton

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Liberty mid-cycle coding sprint

[Ben Pfaff]

On Mon, Apr 06, 2015 at 03:00:17PM -0500, Kyle Mestery wrote:
 I know we're not even at the Liberty Design Summit in Vancouver yet, but I
 wanted to take this time to announce the Neutron mid-cycle coding sprint
 for Liberty. HP has been gracious enough to offer to host at it's Fort
 Collins, CO offices. The dates are set for June 24-26, this is
 Wednesday-Friday. I've got additional information on the etherpad [1].
 
 We'll set the specific agenda in the coming weeks, but the idea is to focus
 on things like the pending neutron-lib work [2] while there, similar to
 what we did with the advanced services split in Utah last year. My
 experience running the past two mid-cycles has been that having these
 earlier in the cycle has been helpful for landing a lot of work near the
 first milestone of a release. I expect this to be the same for Liberty with
 the sprint in Fort Collins.
 
 Please note attendance is not required at all. We will do our best to
 facilitate virtual collaboration for those who cannot travel to the event.
 I wanted to get this out there for folks who have to book travel in advance.

I don't know anything about these events.  Naively: would OVN
development (some of which is in Neutron, much of which is not) be an
appropriate use of time at the event?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] OpenFlow security groups (pre-benchmarking plan)

[Ben Pfaff]

On Tue, Mar 03, 2015 at 09:53:23AM +0100, Miguel Ángel Ajo wrote:
 https://review.openstack.org/#/c/159840/1/doc/source/testing/openflow-firewall.rst
   
 
 I may need some help from the OVS experts to answer the questions from  
 henry.hly.
 
 Ben, Thomas, could you please? (let me know if you are not registered to
 the openstack review system, I could answer in your name).

I added a comment.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] OpenFlow security groups (pre-benchmarking plan)

[Ben Pfaff]

What kind of VLAN support would you need?

On Thu, Feb 26, 2015 at 02:05:41PM -0800, Kevin Benton wrote:
 If OVN chooses not to support VLANs, we will still need the current OVS
 reference anyway so it definitely won't be wasted work.
 
 On Thu, Feb 26, 2015 at 2:56 AM, Miguel Angel Ajo Pelayo 
 majop...@redhat.com wrote:
 
 
  Sharing thoughts that I was having:
 
  May be during the next summit it???s worth discussing the future of the
  reference agent(s), I feel we???ll be replicating a lot of work across
  OVN/OVS/RYU(ofagent) and may be other plugins,
 
  I guess until OVN and it???s integration are ready we can???t stop, so it 
  makes
  sense to keep development at our side, also having an independent plugin
  can help us iterate faster for new features, yet I expect that OVN will be
  more fluent at working with OVS and OpenFlow, as their designers have
  a very deep knowledge of OVS under the hood, and it???s C. ;)
 
  Best regards,
 
  On 26/2/2015, at 7:57, Miguel ??ngel Ajo majop...@redhat.com wrote:
 
  On Thursday, 26 de February de 2015 at 7:48, Miguel ??ngel Ajo wrote:
 
   Inline comments follow after this, but I wanted to respond to Brian
  questionwhich has been cut out:
  We???re talking here of doing a preliminary analysis of the networking
  performance,before writing any real code at neutron level.
 
  If that looks right, then we should go into a preliminary (and orthogonal
  to iptables/LB)implementation. At that moment we will be able to examine
  the scalability of the solutionin regards of switching openflow rules,
  which is going to be severely affectedby the way we use to handle OF rules
  in the bridge:
 * via OpenFlow, making the agent a ???real OF controller, with the
  current effort to use  the ryu framework plugin to do that.   * via
  cmdline (would be alleviated with the current rootwrap work, but the former
  one would be preferred).
  Also, ipset groups can be moved into conjunctive groups in OF (thanks Ben
  Pfaff for theexplanation, if you???re reading this ;-))
  Best,Miguel ??ngel
 
 
  On Wednesday, 25 de February de 2015 at 20:34, Tapio Tallgren wrote:
 
  Hi,
 
  The RFC2544 with near zero packet loss is a pretty standard performance
  benchmark. It is also used in the OPNFV project (
  https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases
  ).
 
  Does this mean that OpenStack will have stateful firewalls (or security
  groups)? Any other ideas planned, like ebtables type filtering?
 
  What I am proposing is in the terms of maintaining the statefulness we
  have nowregards security groups (RELATED/ESTABLISHED connections are
  allowed back on open ports) while adding a new firewall driver working only
  with OVS+OF (no iptables or linux bridge).
 
  That will be possible (without auto-populating OF rules in oposite
  directions) due to
  the new connection tracker functionality to be eventually merged into ovs.
 
 
  -Tapio
 
  On Wed, Feb 25, 2015 at 5:07 PM, Rick Jones rick.jon...@hp.com wrote:
 
  On 02/25/2015 05:52 AM, Miguel ??ngel Ajo wrote:
 
  I???m writing a plan/script to benchmark OVS+OF(CT) vs
  OVS+LB+iptables+ipsets,
  so we can make sure there???s a real difference before jumping into any
  OpenFlow security group filters when we have connection tracking in OVS.
 
  The plan is to keep all of it in a single multicore host, and make
  all the measures within it, to make sure we just measure the
  difference due to the software layers.
 
  Suggestions or ideas on what to measure are welcome, there???s an initial
  draft here:
 
  https://github.com/mangelajo/ovs-experiments/tree/master/ovs-ct
 
 
  Conditions to be benchmarked
 
  Initial connection establishment time
  Max throughput on the same CPU
 
  Large MTUs and stateless offloads can mask a multitude of path-length
  sins.  And there is a great deal more to performance than Mbit/s. While
  some of that may be covered by the first item via the likes of say netperf
  TCP_CRR or TCP_CC testing, I would suggest that in addition to a focus on
  Mbit/s (which I assume is the focus of the second item) there is something
  for packet per second performance.  Something like netperf TCP_RR and
  perhaps aggregate TCP_RR or UDP_RR testing.
 
  Doesn't have to be netperf, that is simply the hammer I wield :)
 
  What follows may be a bit of perfect being the enemy of the good, or
  mission creep...
 
  On the same CPU would certainly simplify things, but it will almost
  certainly exhibit different processor data cache behaviour than actually
  going through a physical network with a multi-core system.  Physical NICs
  will possibly (probably?) have RSS going, which may cause cache lines to be
  pulled around.  The way packets will be buffered will differ as well.  Etc
  etc.  How well the different solutions scale with cores is definitely a
  difference of interest between the two sofware layers.
 
 
 
  Hi rick, thanks for your feedback here, I

Re: [openstack-dev] [neutron] OpenFlow security groups (pre-benchmarking plan)

[Ben Pfaff]

This sounds quite similar to the planned support in OVN to gateway a
logical network to a particular VLAN on a physical port, so perhaps it
will be sufficient.

On Thu, Feb 26, 2015 at 05:58:40PM -0800, Kevin Benton wrote:
 If a port is bound with a VLAN segmentation type, it will get a VLAN id and
 a name of a physical network that it corresponds to. In the current plugin,
 each agent is configured with a mapping between physical networks and OVS
 bridges. The agent takes the bound port information and sets up rules to
 forward traffic from the VM port to the OVS bridge corresponding to the
 physical network. The bridge usually then has a physical interface added to
 it for the tagged traffic to use to reach the rest of the network.
 
 On Thu, Feb 26, 2015 at 4:19 PM, Ben Pfaff b...@nicira.com wrote:
 
  What kind of VLAN support would you need?
 
  On Thu, Feb 26, 2015 at 02:05:41PM -0800, Kevin Benton wrote:
   If OVN chooses not to support VLANs, we will still need the current OVS
   reference anyway so it definitely won't be wasted work.
  
   On Thu, Feb 26, 2015 at 2:56 AM, Miguel Angel Ajo Pelayo 
   majop...@redhat.com wrote:
  
   
Sharing thoughts that I was having:
   
May be during the next summit it???s worth discussing the future of the
reference agent(s), I feel we???ll be replicating a lot of work across
OVN/OVS/RYU(ofagent) and may be other plugins,
   
I guess until OVN and it???s integration are ready we can???t stop, so
  it makes
sense to keep development at our side, also having an independent
  plugin
can help us iterate faster for new features, yet I expect that OVN
  will be
more fluent at working with OVS and OpenFlow, as their designers have
a very deep knowledge of OVS under the hood, and it???s C. ;)
   
Best regards,
   
On 26/2/2015, at 7:57, Miguel ??ngel Ajo majop...@redhat.com wrote:
   
On Thursday, 26 de February de 2015 at 7:48, Miguel ??ngel Ajo wrote:
   
 Inline comments follow after this, but I wanted to respond to Brian
questionwhich has been cut out:
We???re talking here of doing a preliminary analysis of the networking
performance,before writing any real code at neutron level.
   
If that looks right, then we should go into a preliminary (and
  orthogonal
to iptables/LB)implementation. At that moment we will be able to
  examine
the scalability of the solutionin regards of switching openflow rules,
which is going to be severely affectedby the way we use to handle OF
  rules
in the bridge:
   * via OpenFlow, making the agent a ???real OF controller, with the
current effort to use  the ryu framework plugin to do that.   * via
cmdline (would be alleviated with the current rootwrap work, but the
  former
one would be preferred).
Also, ipset groups can be moved into conjunctive groups in OF (thanks
  Ben
Pfaff for theexplanation, if you???re reading this ;-))
Best,Miguel ??ngel
   
   
On Wednesday, 25 de February de 2015 at 20:34, Tapio Tallgren wrote:
   
Hi,
   
The RFC2544 with near zero packet loss is a pretty standard performance
benchmark. It is also used in the OPNFV project (
   
  https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases
).
   
Does this mean that OpenStack will have stateful firewalls (or security
groups)? Any other ideas planned, like ebtables type filtering?
   
What I am proposing is in the terms of maintaining the statefulness we
have nowregards security groups (RELATED/ESTABLISHED connections are
allowed back on open ports) while adding a new firewall driver working
  only
with OVS+OF (no iptables or linux bridge).
   
That will be possible (without auto-populating OF rules in oposite
directions) due to
the new connection tracker functionality to be eventually merged into
  ovs.
   
   
-Tapio
   
On Wed, Feb 25, 2015 at 5:07 PM, Rick Jones rick.jon...@hp.com
  wrote:
   
On 02/25/2015 05:52 AM, Miguel ??ngel Ajo wrote:
   
I???m writing a plan/script to benchmark OVS+OF(CT) vs
OVS+LB+iptables+ipsets,
so we can make sure there???s a real difference before jumping into any
OpenFlow security group filters when we have connection tracking in
  OVS.
   
The plan is to keep all of it in a single multicore host, and make
all the measures within it, to make sure we just measure the
difference due to the software layers.
   
Suggestions or ideas on what to measure are welcome, there???s an
  initial
draft here:
   
https://github.com/mangelajo/ovs-experiments/tree/master/ovs-ct
   
   
Conditions to be benchmarked
   
Initial connection establishment time
Max throughput on the same CPU
   
Large MTUs and stateless offloads can mask a multitude of path-length
sins.  And there is a great deal more to performance than Mbit/s. While
some of that may be covered

Re: [openstack-dev] [neutron] OpenFlow security groups (pre-benchmarking plan)

[Ben Pfaff]

On Thu, Feb 26, 2015 at 07:48:51AM +0100, Miguel Ángel Ajo wrote:
 Also, ipset groups can be moved into conjunctive groups in OF (thanks Ben 
 Pfaff for the
 explanation, if you’re reading this ;-))

You're welcome.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] ML2 versus core plugin for OVN

[Ben Pfaff]

[branching off a discussion on ovs-dev at this point:
http://openvswitch.org/pipermail/dev/2015-February/051609.html]

On Fri, Feb 20, 2015 at 6:56 PM, Kyle Mestery mest...@mestery.com wrote:
 One thing to keep in mind, this ties somewhat into my response to Russell
 earlier on the decision around ML2 vs. core plugin. If we do ML2, there are
 type drivers for VLAN, VXLAN, and GRE tunnels. There is no TypeDriver for
 STT tunnels upstream now. It's just an item we need on the TODO list if we
 go down the STT tunnel path.

It was suggested to me off-list that this part of the discussion should be on
openstack-dev, so here it is ;-)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ovs-dev] [PATCH 8/8] [RFC] [neutron] ovn: Start work on design ocumentation.

[Ben Pfaff]

On Fri, Feb 20, 2015 at 12:45:46PM +0100, Miguel Ángel Ajo wrote:
 On Thursday, 19 de February de 2015 at 23:15, Kyle Mestery wrote:  
  On Thu, Feb 19, 2015 at 3:55 PM, Ben Pfaff b...@nicira.com 
  (mailto:b...@nicira.com) wrote:
   My initial reaction is that we can implement security groups as
   another action in the ACL table that is similar to allow but in
   addition permits reciprocal inbound traffic.  Does that sound
   sufficient to you?
 Yes, having fine grained allows (matching on protocols, ports, and
 remote ips would satisfy the neutron use case).
 
 Also we use connection tracking to allow reciprocal inbound traffic
 via ESTABLISHED/RELATED, any equivalent solution would do.
 
 For reference, our SG implementation, currently is able to match on
 combinations of:
 
 * direction: ingress/egress
 * protocol: icmp/tcp/udp/raw number
 * port_range:  min-max   (it’s always dst)
 * L2 packet ethertype: IPv4, IPv6, etc...
 * remote_ip_prefix: as a CIDR   or* remote_group_id (to reference all 
 other IPs in a certain group)
 
 All of them assume connection tracking so known connection packets will
 go the other way around.

OK.  All of those should work OK.  (I don't know for sure whether we'll
have explicit groups; initially, probably not.)

   Is the exponential explosion due to cross-producting, that is, because
   you have, say, n1 source addresses and n2 destination addresses and so
   you need n1*n2 flows to specify all the combinations?  We aim to solve
   that in OVN by giving the CMS direct support for more sophisticated
   matching rules, so that it can say something like:

   ip.src in {a, b, c, ...}  ip.dst in {d, e, f, ...}
(tcp.src in {80, 443, 8080} || tcp.dst in {80, 443, 8080})
 
 That sounds good and very flexible.

   and let OVN implement it in OVS via the conjunctive match feature
   recently added, which is like a set membership match but more
   powerful.  
 Hmm, where can I find examples about that feature, sounds interesting.

If you look at ovs-ofctl(8) in a development version of OVS, such as
http://benpfaff.org/~blp/dist-docs/ovs-ofctl.8.pdf
search for conjunction, which explains the implementation.  (This
isn't the form that Neutron would use with OVN; that is the Boolean
expression syntax above.)

   It might still be nice to support lists of IPs (or
   whatever), since these lists could still recur in a number of
   circumstances, but my guess is that this will help a lot even without
   that.

 As afar as I understood, given the way megaflows resolve rules via hashes
 even if we had lots of rules with different ip addresses, that would be very 
 fast,
 probably as fast or more than our current ipset solution.
 
 The only caveat would be having to update lots of flow rules when a port goes
 in or out of a security group, since you have to go and clear/add the rules 
 to each
 single port on the same security group (as long as they have 1 rule 
 referencing the sg).

That sounds like another good argument for allowing explicit groups.  I
have a design in mind for that but I doubt it's the first thing to
implement.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ovs-dev] [PATCH 8/8] [RFC] ovn: Start work on design ocumentation.

[Ben Pfaff]

[moving this conversation to openstack-dev because it's more
interesting there and that avoids crossposting to a subscribers-only
list]

On Thu, Feb 19, 2015 at 10:57:02AM +0100, Miguel ??ngel Ajo wrote:
I specially liked the VIF port lifecycle, looks good to me, Ionly miss 
 some  
 ???port_security??? concepts we have in neutron, which I guess could have 
 been  
 deliberately omitted for a start.
 
In neutron we have something called security groups, and every port
 belongs to 1 or more security groups.  Each security group has a list of
 rules to control traffic at port level in a very fine grained fashion 
 (ingress/egress
 protocol/flags/etc???   remote_ip/mask or security_group ID)
 
 I guess we could build  render security_group ID to multiple IPs for each 
 port,
 but then we will miss the ingress/egress and protocol flags (like type  of 
 protocol,
 ports, etc.. [1])
 
 Also, be aware, that not having security group ID references from neutron,
 when lot???s of ports go to the same security group we end up with an 
 exponential
 growth of rules / OF entries per port, we solved this in the server-agent
 communication for the reference OVS solution by keeping a lists of IPs  
 belonging to security group IDs, and then, separately having the  
 references from the rules.

Thanks a lot for the comment.

We aim to fully support security groups in OVN.  The current documents
don't capture that intent.  (That's partly because we're planning to
implement them in terms of some new connection tracking code for OVS
that's still in the process of getting committed, and partly because I
haven't fully thought them through yet.)

My initial reaction is that we can implement security groups as
another action in the ACL table that is similar to allow but in
addition permits reciprocal inbound traffic.  Does that sound
sufficient to you?

Is the exponential explosion due to cross-producting, that is, because
you have, say, n1 source addresses and n2 destination addresses and so
you need n1*n2 flows to specify all the combinations?  We aim to solve
that in OVN by giving the CMS direct support for more sophisticated
matching rules, so that it can say something like:

ip.src in {a, b, c, ...}  ip.dst in {d, e, f, ...}
 (tcp.src in {80, 443, 8080} || tcp.dst in {80, 443, 8080})

and let OVN implement it in OVS via the conjunctive match feature
recently added, which is like a set membership match but more
powerful.  It might still be nice to support lists of IPs (or
whatever), since these lists could still recur in a number of
circumstances, but my guess is that this will help a lot even without
that.

Thoughts?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron]??[ovs-dev] [PATCH 8/8] [RFC] ovn: Start work on design Documentation.

[Ben Pfaff]

Thanks.  It looks like openstack-dev only accepts posts from
subscribers, so I've followed up to your more detailed feedback only
on that list.  For anyone following along at home, here's an archives
link:
http://lists.openstack.org/pipermail/openstack-dev/2015-February/057376.html

On Thu, Feb 19, 2015 at 10:38:03AM +0100, Miguel ??ngel Ajo wrote:
 Thank you Ben!,  
 
 Cross posting [1] to openstack list /neutron.
 
 
 [1] http://benpfaff.org/~blp/dist-docs. 
 
 On Thursday, 19 de February de 2015 at 09:13, Ben Pfaff wrote:
 
  On Thu, Feb 19, 2015 at 12:12:26AM -0800, Ben Pfaff wrote:
   This commit adds preliminary design documentation for Open Virtual 
   Network,
   or OVN, a new OVS-based project to add support for virtual networking to
   OVS, initially with OpenStack integration.
   
   This initial design has been influenced by many people, including (in
   alphabetical order) Aaron Rosen, Chris Wright, Jeremy Stribling,
   Justin Pettit, Ken Duda, Madhu Venugopal, Martin Casado, Pankaj Thakkar,
   Russell Bryant, and Teemu Koponen. All blunders, however, are due to my
   own hubris.
   
   Signed-off-by: Ben Pfaff b...@nicira.com (mailto:b...@nicira.com)
  
  I've posted the rendered version of the documentation following this
  commit at http://benpfaff.org/~blp/dist-docs. You probably want to look
  at the ovn* manpages, especially ovn-architecture(7), ovn(5), and
  ovn-nb(5).
  ___
  dev mailing list
  d...@openvswitch.org (mailto:d...@openvswitch.org)
  http://openvswitch.org/mailman/listinfo/dev
  
  
 
 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ovs-dev] [PATCH 8/8] [RFC] [neutron] ovn: Start work on design ocumentation.

[Ben Pfaff]

Thanks for both of those bits of information.

I've added an allow-related action for v3.

On Thu, Feb 19, 2015 at 06:38:46PM -0800, Kevin Benton wrote:
 Yes, if there is an action similar to the 'RELATED' and 'ESTABLISHED'
 keywords, then that should be adequate to replace the iptables solution
 that we have now.
 
 You are correct about the reason behind the explosion of rules. In security
 groups, we allow the source/dest field to be a reference to a security
 group. That was resulting in one security group rule turning into N
 iptables rules, where N was the number of IPs in the security group. This
 was resolved using IP sets. If OVN supports the set syntax you described,
 that should achieve parity on that front as well.
 
 
 On Thu, Feb 19, 2015 at 2:15 PM, Kyle Mestery mest...@mestery.com wrote:
 
  [Adding neutron tag to subject, comments below.]
 
  On Thu, Feb 19, 2015 at 3:55 PM, Ben Pfaff b...@nicira.com wrote:
 
  [moving this conversation to openstack-dev because it's more
  interesting there and that avoids crossposting to a subscribers-only
  list]
 
  On Thu, Feb 19, 2015 at 10:57:02AM +0100, Miguel ??ngel Ajo wrote:
  I specially liked the VIF port lifecycle, looks good to me, Ionly
  miss some
   ???port_security??? concepts we have in neutron, which I guess could
  have been
   deliberately omitted for a start.
  
  In neutron we have something called security groups, and every port
   belongs to 1 or more security groups.  Each security group has a list of
   rules to control traffic at port level in a very fine grained fashion
  (ingress/egress
   protocol/flags/etc???   remote_ip/mask or security_group ID)
  
   I guess we could build  render security_group ID to multiple IPs for
  each port,
   but then we will miss the ingress/egress and protocol flags (like type
  of protocol,
   ports, etc.. [1])
  
   Also, be aware, that not having security group ID references from
  neutron,
   when lot???s of ports go to the same security group we end up with an
  exponential
   growth of rules / OF entries per port, we solved this in the
  server-agent
   communication for the reference OVS solution by keeping a lists of IPs
   belonging to security group IDs, and then, separately having the
   references from the rules.
 
  Thanks a lot for the comment.
 
  We aim to fully support security groups in OVN.  The current documents
  don't capture that intent.  (That's partly because we're planning to
  implement them in terms of some new connection tracking code for OVS
  that's still in the process of getting committed, and partly because I
  haven't fully thought them through yet.)
 
  My initial reaction is that we can implement security groups as
  another action in the ACL table that is similar to allow but in
  addition permits reciprocal inbound traffic.  Does that sound
  sufficient to you?
 
  Is the exponential explosion due to cross-producting, that is, because
  you have, say, n1 source addresses and n2 destination addresses and so
  you need n1*n2 flows to specify all the combinations?  We aim to solve
  that in OVN by giving the CMS direct support for more sophisticated
  matching rules, so that it can say something like:
 
  ip.src in {a, b, c, ...}  ip.dst in {d, e, f, ...}
   (tcp.src in {80, 443, 8080} || tcp.dst in {80, 443, 8080})
 
  and let OVN implement it in OVS via the conjunctive match feature
  recently added, which is like a set membership match but more
  powerful.  It might still be nice to support lists of IPs (or
  whatever), since these lists could still recur in a number of
  circumstances, but my guess is that this will help a lot even without
  that.
 
  Thoughts?
 
  This all sounds really good to me Ben. I look forward to seeing the
  connection tracking code land and some design details on the security
  groups aspects of OVN published once that happens!
 
  Thanks,
  Kyle
 
 
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
  openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 -- 
 Kevin Benton

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev