On Thu, Aug 7, 2014 at 12:08 PM, Kevin Benton <[email protected]> wrote:
> >I mean't 'side stepping' why GBP allows for the comment you made > previous, "With the latter, a mapping driver could determine that > communication between these two hosts can be prevented by using an ACL on a > router or a switch, which doesn't violate the user's intent and buys a > performance improvement and works with ports that don't support security > groups.". > > >Neutron's current API is a logical abstraction and enforcement can be > done however one chooses to implement it. I'm really trying to understand > at the network level why GBP allows for these optimizations and performance > improvements you talked about. > > You absolutely cannot enforce security groups on a firewall/router that > sits at the boundary between networks. If you try, you are lying to the > end-user because it's not enforced at the port level. The current neutron > APIs force you to decide where things like that are implemented. > The current neutron API's are just logical abstractions. Where and how things are actually enforced are 100% an implementation detail of a vendors system. Anyways, moving the discussion to the etherpad... > > The higher level abstractions give you the freedom to move the enforcement > by allowing the expression of broad connectivity requirements. > >Why are you bringing up logging connections? > > This was brought up as a feature proposal to FWaaS because this is a basic > firewall feature missing from OpenStack. However, this does not preclude a > FWaaS vendor from logging. > > >Personally, I think one could easily write up a very short document > probably less than one page with examples showing/exampling how the current > neutron API works even without a much networking background. > > The difficulty of the API for establishing basic connectivity isn't really > the problem. It's when you have to compose a bunch of requirements and make > sure nothing is violating auditing and connectivity constraints that it > becomes a problem. We are arguing about the levels of abstraction. You > could also write up a short document explaining to novice programmers how > to use C to read and write database entries to an sqlite database, but that > doesn't mean it's the best level of abstraction for what the users are > trying to accomplish. > > I'll let someone else explain the current GBP API because I'm not working > on that. I'm just trying to convince you of the value of declarative > network configuration. > > > On Thu, Aug 7, 2014 at 12:02 PM, Aaron Rosen <[email protected]> > wrote: > >> >> >> >> On Thu, Aug 7, 2014 at 9:54 AM, Kevin Benton <[email protected]> wrote: >> >>> You said you had no idea what group based policy was buying us so I >>> tried to illustrate what the difference between declarative and imperative >>> network configuration looks like. That's the major selling point of GBP so >>> I'm not sure how that's 'side stepping' any points. It removes the need for >>> the user to pick between implementation details like security >>> groups/FWaaS/ACLs. >>> >> >> I mean't 'side stepping' why GBP allows for the comment you made >> previous, "With the latter, a mapping driver could determine that >> communication between these two hosts can be prevented by using an ACL on a >> router or a switch, which doesn't violate the user's intent and buys a >> performance improvement and works with ports that don't support security >> groups.". >> >> Neutron's current API is a logical abstraction and enforcement can be >> done however one chooses to implement it. I'm really trying to understand >> at the network level why GBP allows for these optimizations and performance >> improvements you talked about. >> >> >> >>> >So are you saying that GBP allows someone to be able to configure an >>> application that at the end of the day is equivalent to >>> networks/router/FWaaS rules without understanding networking concepts? >>> >>> It's one thing to understand the ports an application leverages and >>> another to understand the differences between configuring VM firewalls, >>> security groups, FWaaS, and router ACLs. >>> >> >> Sure, but how does group based policy solve this. Security Groups and >> FWaaS are just different places of enforcement. Say I want different >> security enforcement on my router than on my instances. One still needs to >> know enough to tell group based policy this right? They need to know >> enough that there are different enforcement points? How is doing this with >> Group based policy make it easier? >> >> >> >>> > I'm also curious how this GBP is really less error prone than the >>> model we have today as it seems the user will basically have to tell >>> neutron the same information about how he wants his networking to function. >>> >>> With GBP, the user just gives the desired end result (e.g. allow >>> connectivity between endpoint groups via TCP port 22 with all connections >>> logged). Without it, the user has to do the following: >>> >> >> Why are you bringing up logging connections? Neutron has no concept of >> this at all today in it's code base. Is logging something related to GBP? >> >>> >>> 1. create a network/subnet for each endpoint group >>> 2. allow all traffic on the security groups since the logging would >>> need to be accomplished with FWaaS >>> 3. create an FWaaS instance >>> 4. attach the FWaaS to both networks >>> >>> Today FWaaS api is still incomplete as there is no real point of >> enforcement in it's api (though it really seems that it should just be >> router ports) it's just global on the router right now. >> >> >>> >>> 1. add an FWaaS policy and the FWaaS rules to allow the correct >>> traffic >>> >>> I'd more or less agree with these steps. Would you mind also giving the >> steps involved in group based policy so we can compare? >> >> >> >>> The declarative approach is less error prone because the user can give >>> neutron the desired state of connectivity rather than a mentally compiled >>> set of instructions describing how to configure a bunch of individual >>> network components. How well do you think someone will handle the latter >>> approach that got all of their networking knowledge from one college course >>> 5 years ago? >>> >> >> Personally, I think one could easily write up a very short document >> probably less than one page with examples showing/exampling how the current >> neutron API works even without a much networking background. The funny >> thing is I've been trying to completely understand the proposed group >> policy api for a little while now and I'm still having trouble. Seems like >> we're taking abstractions that are quite well known/understood and changing >> them to a different model that requires one to know about this new >> terminology: >> >> >> Endpoint (EP): An L2/L3 addressable entity. >> Endpoint Group (EPG): A collection of endpoints. >> Contract: It defines how the application services provided by an EPG can >> be accessed. In effect it specifies how an EPG communicates with other >> EPGs. A Contract consists of Policy Rules. >> Policy Rule: These are individual rules used to define the communication >> criteria between EPGs. Each rule contains a Filter, Classifier, and Action. >> Classifier: Characterizes the traffic that a particular Policy Rule acts >> on. Corresponding action is taken on traffic that satisfies this >> classification criteria. >> Action: The action that is taken for a matching Policy Rule defined in a >> Contract. >> Filter: Provides a way to tag a Policy Rule with Capability and Role >> labels. >> Capability: It is a Policy Label that defines what part of a Contract a >> particular EPG provides. >> Role: It is a Policy Label that defines what part of a Contract an EPG >> wants to consume. >> Contract Scope: An EPG conveys its intent to provide or consume a >> Contract (or its part) by defining a Contract Scope which references the >> target Contract. >> Selector: A Contract Scope can define additional constraints around >> choosing the matching provider or consumer EPGs for a Contract via a >> Selector. >> Policy Labels: These are labels contained within a namespace hierarchy >> and used to define Capability and Role tags used in Filters. >> Bridge Domain: Used to define a L2 boundary and impose additional >> constraints (such as no broadcast) within that L2 boundary. >> Routing Domain: Used to define a non-overlapping IP address space. >> >> >> >>> >>> IIRC one of the early nova parity requests was an API to automagically >>> setup a neutron network and router so no extra work would be required to >>> get instances connected to the Internet. That wasn't requested because >>> people thought neutron networks were too easy to setup already. :-) >>> >> >> I think the confusion why that comment was made is probably because the >> nova-networks model doesn't have a concept of routers. Neutron can operate >> in the same way though with flat and provider networks. Either way one >> could easily write a nova-binding to the command they used previously to >> create a router and uplink it hiding the fact that there is a router there. >> >>> >>> >>> On Thu, Aug 7, 2014 at 9:10 AM, Aaron Rosen <[email protected]> >>> wrote: >>> >>>> Hi Kevin, >>>> >>>> I feel as your latest response is completely side stepping the points >>>> we have been trying to get to in the last series of emails. At the end of >>>> the day I don't believe we are changing the laws of networking (or perhaps >>>> we are?). Thus I think it's important to actually get down to the >>>> networking level to actually figure out why optimizations such as this one >>>> are enabled via GBP and not the model we have: >>>> >>>> >>>> "With the latter, a mapping driver could determine that communication >>>> between these two hosts can be prevented by using an ACL on a router or a >>>> switch, which doesn't violate the user's intent and buys a performance >>>> improvement and works with ports that don't support security groups." >>>> >>>> >>>> >>>> On Wed, Aug 6, 2014 at 7:48 PM, Kevin Benton <[email protected]> wrote: >>>> >>>>> Do you not see a difference between explicitly configuring networks, a >>>>> router and FWaaS rules with logging and just stating that two groups of >>>>> servers can only communicate via one TCP port with all connections logged? >>>>> The first is very prone to errors for someone deploying an application >>>>> without a strong networking background, and the latter is basically just >>>>> stating the requirements and letting Neutron figure out how to implement >>>>> it. >>>>> >>>>> So are you saying that GBP allows someone to be able to configure an >>>> application that at the end of the day is equivalent to >>>> networks/router/FWaaS rules without understanding networking concepts? I'm >>>> also curious how this GBP is really less error prone than the model we have >>>> today as it seems the user will basically have to tell neutron the same >>>> information about how he wants his networking to function. >>>> >>>> >>>> >>>>> Just stating requirements becomes even more important when something >>>>> like the logging requirement comes from someone other than the app >>>>> deployer >>>>> (e.g. a security team). In the above example, someone could set everything >>>>> up using security groups; however, when the logging requirement came in >>>>> from the security team, they would have to undo all of that work and >>>>> replace it with something like FWaaS that can centrally log all of the >>>>> connections. >>>>> >>>>> It's the difference between using puppet and bash scripts. Sure you >>>>> can write a script that uses awk/sed to ensure that an ini file has a >>>>> particular setting and then restart a service if the setting is changed, >>>>> but it's much easier and less error prone to just write a puppet manifest >>>>> that uses the INI module with a pointer to the file, the section name, the >>>>> key, and the value with a notification to restart the service. >>>>> >>>>> >>>>> >>>>> On Wed, Aug 6, 2014 at 7:40 PM, Aaron Rosen <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> On Wed, Aug 6, 2014 at 5:27 PM, Kevin Benton <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Web tier can communicate with anything except for the DB. >>>>>>> App tier can only communicate with Web and DB. >>>>>>> DB can communicate with App. >>>>>>> >>>>>>> These high-level constraints can then be implemented as security >>>>>>> groups like you showed, or network hardware ACLs like I had shown. >>>>>>> But if you start with the security groups API, you are forcing it to >>>>>>> be implemented there. >>>>>>> >>>>>>> >>>>>> I still have no idea what group based policy is buying us then. It >>>>>> seems to me that the key point we've identified going backing and forth >>>>>> here is the difference between the current model and the GBP model is >>>>>> that >>>>>> GBP constricts topology which allows us to write these types of >>>>>> enforcement >>>>>> rules. The reason we want this is because it yields performance >>>>>> optimizations (for some reason, which I don't think we've gotten into). >>>>>> Would you agree this is accurate? >>>>>> >>>>>> Honestly, I know a lot of work has been put into this. I haven't said >>>>>> I'm for or against it either. I'm really just trying to understand what >>>>>> is >>>>>> the motivation for this and why does it make neutron better. >>>>>> >>>>>> Best, >>>>>> >>>>>> Aaron >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> On Wed, Aug 6, 2014 at 6:06 PM, Aaron Rosen <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Aug 6, 2014 at 4:46 PM, Kevin Benton <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> That's the point. By using security groups, you are forcing a >>>>>>>>> certain kind of enforcement that must be honored and might not be >>>>>>>>> necessary >>>>>>>>> if the original intent was just to isolate between groups. In the >>>>>>>>> example >>>>>>>>> you gave, it cannot be implemented on the router without violating the >>>>>>>>> constraints of the security group. >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Kevin, >>>>>>>> >>>>>>>> Mind proposing an alternative example then. The only way I can see >>>>>>>> this claim to be made is because Group Based policy is actually >>>>>>>> limiting >>>>>>>> what a tenants desired topology can be? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Aaron >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Wed, Aug 6, 2014 at 5:39 PM, Aaron Rosen < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Aug 6, 2014 at 4:18 PM, Kevin Benton <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> >Given this information I don't see any reason why the backend >>>>>>>>>>> system couldn't do enforcement at the logical router and if it did >>>>>>>>>>> so >>>>>>>>>>> neither parties would know. >>>>>>>>>>> >>>>>>>>>>> With security groups you are specifying that nothing can contact >>>>>>>>>>> these devices on those ports unless they come from the allowed IP >>>>>>>>>>> addresses. If you tried to enforce this at the router you would be >>>>>>>>>>> violating that specification because devices in the same subnet >>>>>>>>>>> would be >>>>>>>>>>> able to communicate on those blocked ports. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Sure, though this is a problem of where you are doing your >>>>>>>>>> enforcement. If the backend system chooses to implement this >>>>>>>>>> optimization >>>>>>>>>> in this fashion (which was the example you gave previously [1]). >>>>>>>>>> Then, if >>>>>>>>>> the topology changes, i.e adding a port to the same network with >>>>>>>>>> conflicting security group rules, the backend system can no longer >>>>>>>>>> optimize >>>>>>>>>> in this same fashion at the router level and a more fine grain >>>>>>>>>> filtering >>>>>>>>>> will need to be done. How would this be any different with group >>>>>>>>>> based >>>>>>>>>> policy? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [1] - With the latter, a mapping driver could determine that >>>>>>>>>> communication between these two hosts can be prevented by using an >>>>>>>>>> ACL on a >>>>>>>>>> router or a switch, which doesn't violate the user's intent and buys >>>>>>>>>> a >>>>>>>>>> performance improvement and works with ports that don't support >>>>>>>>>> security >>>>>>>>>> groups. >>>>>>>>>> >>>>>>>>>> states >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Aug 6, 2014 at 5:00 PM, Aaron Rosen < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Aug 6, 2014 at 3:35 PM, Kevin Benton <[email protected] >>>>>>>>>>>> > wrote: >>>>>>>>>>>> >>>>>>>>>>>>> By working at the port level you have already eliminated your >>>>>>>>>>>>> ability to implement the filtering at different components of the >>>>>>>>>>>>> network. >>>>>>>>>>>>> They now need to be implemented in stateful rules at the port >>>>>>>>>>>>> level and the >>>>>>>>>>>>> device has to support security groups. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Lets take this example where we setup a 2 tier app with >>>>>>>>>>>> web-servers and db-servers that are connected on two different >>>>>>>>>>>> networks >>>>>>>>>>>> attached to a router. We add a security group rules such that >>>>>>>>>>>> web-servers >>>>>>>>>>>> can talk to db-servers on tcp:3306 and a rule to allow tcp:80 into >>>>>>>>>>>> the >>>>>>>>>>>> web-servers from anywhere. >>>>>>>>>>>> >>>>>>>>>>>> neutron net-create web_net >>>>>>>>>>>> neutron subnet-create --name web_subnet web_net 10.0.0.0/24 >>>>>>>>>>>> >>>>>>>>>>>> neutron net-create db_net >>>>>>>>>>>> neutron subnet-create --name db_subnet db_net 10.2.0.0/24 >>>>>>>>>>>> >>>>>>>>>>>> neutron router-create myrouter >>>>>>>>>>>> neutron router-interface-add myrouter web_subnet >>>>>>>>>>>> neutron router-interface-add myrouter db_subnet >>>>>>>>>>>> >>>>>>>>>>>> neutron security-group-create web-servers; >>>>>>>>>>>> neutron security-group-create db-servers; >>>>>>>>>>>> >>>>>>>>>>>> # add rule to allow web members to talk to the db-servers on >>>>>>>>>>>> TCP 3306 for their db connection; >>>>>>>>>>>> neutron security-group-rule-create --protocol TCP >>>>>>>>>>>> --port-range-min 3306 --port-range-max 3306 --remote-group-id >>>>>>>>>>>> web-servers >>>>>>>>>>>> db-servers >>>>>>>>>>>> >>>>>>>>>>>> # add rule to allow TCP 80 into the web-server sg >>>>>>>>>>>> neutron security-group-rule-create --protocol TCP >>>>>>>>>>>> --port-range-min 80 --port-range-max 80 web-servers db-servers >>>>>>>>>>>> >>>>>>>>>>>> # create some ports with desired security profiles. >>>>>>>>>>>> neutron port-create --security-group web-servers web_net >>>>>>>>>>>> neutron port-create --security-group web-servers web_net >>>>>>>>>>>> >>>>>>>>>>>> neutron port-create --security-group db-servers db_net >>>>>>>>>>>> neutron port-create --security-group db-servers db_net >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Now to your point: >>>>>>>>>>>> >>>>>>>>>>>> By working at the port level you have already eliminated your >>>>>>>>>>>>> ability to implement the filtering at different components of the >>>>>>>>>>>>> network. >>>>>>>>>>>>> They now need to be implemented in stateful rules at the port >>>>>>>>>>>>> level and the >>>>>>>>>>>>> device has to support security groups. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> Given this information I don't see any reason why the backend >>>>>>>>>>>> system couldn't do enforcement at the logical router and if it did >>>>>>>>>>>> so >>>>>>>>>>>> neither parties would know. The backend system should have the >>>>>>>>>>>> full graph >>>>>>>>>>>> of everything and be able to do enforcement optimizations where >>>>>>>>>>>> ever it >>>>>>>>>>>> likes. >>>>>>>>>>>> >>>>>>>>>>>> btw: I say the enforcement could be done on the logical router >>>>>>>>>>>> though the backend system could also do this on the physical fabic >>>>>>>>>>>> as well >>>>>>>>>>>> if it wanted to as it should also know that graph. No? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Aug 6, 2014 at 4:03 PM, Aaron Rosen < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Aug 6, 2014 at 12:46 PM, Kevin Benton < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >I believe the referential security group rules solve this >>>>>>>>>>>>>>> problem (unless I'm not understanding): >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think the disconnect is that you are comparing the way to >>>>>>>>>>>>>>> current mapping driver implements things for the reference >>>>>>>>>>>>>>> implementation >>>>>>>>>>>>>>> with the existing APIs. Under this light, it's not going to >>>>>>>>>>>>>>> look like there >>>>>>>>>>>>>>> is a point to this code being in Neutron since, as you said, the >>>>>>>>>>>>>>> abstraction could happen at a client. However, this changes >>>>>>>>>>>>>>> once new >>>>>>>>>>>>>>> mapping drivers can be added that implement things differently. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let's take the security groups example. Using the security >>>>>>>>>>>>>>> groups API directly is imperative ("put a firewall rule on this >>>>>>>>>>>>>>> port that >>>>>>>>>>>>>>> blocks this IP") compared to a higher level declarative >>>>>>>>>>>>>>> abstraction ("make >>>>>>>>>>>>>>> sure these two endpoints cannot communicate"). With the former, >>>>>>>>>>>>>>> the ports >>>>>>>>>>>>>>> must support security groups and there is nowhere except for >>>>>>>>>>>>>>> the firewall >>>>>>>>>>>>>>> rules on that port to implement it without violating the user's >>>>>>>>>>>>>>> expectation. With the latter, a mapping driver could determine >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>> communication between these two hosts can be prevented by using >>>>>>>>>>>>>>> an ACL on a >>>>>>>>>>>>>>> router or a switch, which doesn't violate the user's intent and >>>>>>>>>>>>>>> buys a >>>>>>>>>>>>>>> performance improvement and works with ports that don't support >>>>>>>>>>>>>>> security >>>>>>>>>>>>>>> groups. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Group based policy is trying to move the requests into the >>>>>>>>>>>>>>> declarative abstraction so optimizations like the one above can >>>>>>>>>>>>>>> be made. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Kevin, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Interesting points. Though, let me ask this. Why do we need >>>>>>>>>>>>>> to move to a declarative API abstraction in neutron in order to >>>>>>>>>>>>>> perform >>>>>>>>>>>>>> this optimization on the backend? For example, In the current >>>>>>>>>>>>>> neutron model >>>>>>>>>>>>>> say we want to create a port with a security group attached to >>>>>>>>>>>>>> it called >>>>>>>>>>>>>> web that allows TCP:80 in and members who are in a security >>>>>>>>>>>>>> group called >>>>>>>>>>>>>> database. From this mapping I fail to see how it's really any >>>>>>>>>>>>>> different >>>>>>>>>>>>>> from the declarative model? The ports in neutron are logical >>>>>>>>>>>>>> abstractions >>>>>>>>>>>>>> and the backend system could be implemented in order to >>>>>>>>>>>>>> determine that the >>>>>>>>>>>>>> communication between these two hosts could be prevented by >>>>>>>>>>>>>> using an ACL on >>>>>>>>>>>>>> a router or switch as well. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Aaron >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> OpenStack-dev mailing list >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> OpenStack-dev mailing list >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Kevin Benton >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> OpenStack-dev mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> >>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> OpenStack-dev mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> >>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Kevin Benton >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> OpenStack-dev mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> OpenStack-dev mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Kevin Benton >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> OpenStack-dev mailing list >>>>>>>>> [email protected] >>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> OpenStack-dev mailing list >>>>>>>> [email protected] >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Kevin Benton >>>>>>> >>>>>>> _______________________________________________ >>>>>>> OpenStack-dev mailing list >>>>>>> [email protected] >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> OpenStack-dev mailing list >>>>>> [email protected] >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Kevin Benton >>>>> >>>>> _______________________________________________ >>>>> OpenStack-dev mailing list >>>>> [email protected] >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> OpenStack-dev mailing list >>>> [email protected] >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>> >>> >>> -- >>> Kevin Benton >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> [email protected] >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> _______________________________________________ >> OpenStack-dev mailing list >> [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > > -- > Kevin Benton > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
