Nice job! That's awesome. Thanks, Édouard.
On Thu, Aug 21, 2014 at 8:02 AM, Miguel Angel Ajo Pelayo < [email protected]> wrote: > Thank you shihanzhang!, > > I can't believe I didn't realize the ipset part spec was accepted I live > on my own bubble... I will be reviewing and testing/helping on that part > too during the next few days, I was too concentrated in the RPC part. > > > Best regards, > > ----- Original Message ----- > > hi neutroner! > > my patch about BP: > > > https://blueprints.launchpad.net/openstack/?searchtext=add-ipset-to-security > > need install ipset in devstack, I have commit the patch: > > https://review.openstack.org/#/c/113453/, who can help me review it, > thanks > > very much! > > > > Best regards, > > shihanzhang > > > > > > > > > > At 2014-08-21 10:47:59, "Martinx - ジェームズ" <[email protected]> > wrote: > > > > > > > > +1 "NFTablesDriver"! > > > > Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example: > > https://home.regit.org/2014/02/suricata-and-nftables/ > > > > Then, I'm wondering here... What benefits might come for OpenStack Nova / > > Neutron, if it comes with a NFTables driver, instead of the current > > IPTables?! > > > > * E fficient Security Group design? > > * Better FWaaS, maybe with NAT(44/66) support? > > * Native support for IPv6, with the defamed NAT66 built-in, simpler > "Floating > > IP" implementation, for both v4 and v6 networks under a single > > implementation ( I don't like NAT66, I prefer a `routed Floating IPv6` > > version ) ? > > * Metadata over IPv6 still using NAT(66) ( I don't like NAT66 ), single > > implementation? > > * Suricata-as-a-Service?! > > > > It sounds pretty cool! :-) > > > > > > On 20 August 2014 23:16, Baohua Yang < [email protected] > wrote: > > > > > > > > Great！ > > We met similar problems. > > The current mechanisms produce too many iptables rules, and it's hard to > > debug. > > Really look forward to seeing a more efficient security group design. > > > > > > On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery < > [email protected] > > > wrote: > > > > > > > > On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang < [email protected] > > wrote: > > > > > > With the deployment 'nova + neutron + openvswitch', when we bulk create > > > about 500 VM with a default security group, the CPU usage of > neutron-server > > > and openvswitch agent is very high, especially the CPU usage of > openvswitch > > > agent will be 100%, this will cause creating VMs failed. > > > > > > With the method discussed in mailist: > > > > > > 1) ipset optimization ( https://review.openstack.org/#/c/100761/ ) > > > > > > 3) sg rpc optimization (with fanout) > > > ( https://review.openstack.org/#/c/104522/ ) > > > > > > I have implement these two scheme in my deployment, when we again bulk > > > create about 500 VM with a default security group, the CPU usage of > > > openvswitch agent will reduce to 10%, even lower than 10%, so I think > the > > > iprovement of these two options are very efficient. > > > > > > Who can help us to review our spec? > > > > > This is great work! These are on my list of things to review in detail > > soon, but given the Neutron sprint this week, I haven't had time yet. > > I'll try to remedy that by the weekend. > > > > Thanks! > > Kyle > > > > > Best regards, > > > shihanzhang > > > > > > > > > > > > > > > > > > At 2014-07-03 10:08:21, "Ihar Hrachyshka" < [email protected] > > wrote: > > >>-----BEGIN PGP SIGNED MESSAGE----- > > >>Hash: SHA512 > > >> > > >>Oh, so you have the enhancement implemented? Great! Any numbers that > > >>shows how much we gain from that? > > >> > > >>/Ihar > > >> > > >>On 03/07/14 02:49, shihanzhang wrote: > > >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today > > >>> I will modify my spec, when the spec is approved, I will commit the > > >>> codes as soon as possilbe! > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" < [email protected] > > > >>> wrote: > > >>>> > > >>>> Nice Shihanzhang, > > >>>> > > >>>> Do you mean the ipset implementation is ready, or just the > > >>>> spec?. > > >>>> > > >>>> > > >>>> For the SG group refactor, I don't worry about who does it, or > > >>>> who takes the credit, but I believe it's important we address > > >>>> this bottleneck during Juno trying to match nova's scalability. > > >>>> > > >>>> Best regards, Miguel Ángel. > > >>>> > > >>>> > > >>>> On 07/02/2014 02:50 PM, shihanzhang wrote: > > >>>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that > > >>>>> split the work in several specs, I have finished the work ( > > >>>>> ipset optimization), you can do 'sg rpc optimization (without > > >>>>> fanout)'. as the third part(sg rpc optimization (with fanout)), > > >>>>> I think we need talk about it, because just using ipset to > > >>>>> optimize security group agent codes does not bring the best > > >>>>> results! > > >>>>> > > >>>>> Best regards, shihanzhang. > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" < [email protected] > > > >>>>> wrote: > > >>> On 02/07/14 10:12, Miguel Angel Ajo wrote: > > >>> > > >>>> Shihazhang, > > >>> > > >>>> I really believe we need the RPC refactor done for this cycle, > > >>>> and given the close deadlines we have (July 10 for spec > > >>>> submission and July 20 for spec approval). > > >>> > > >>>> Don't you think it's going to be better to split the work in > > >>>> several specs? > > >>> > > >>>> 1) ipset optimization (you) 2) sg rpc optimization (without > > >>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you > > >>>> , me) > > >>> > > >>> > > >>>> This way we increase the chances of having part of this for the > > >>>> Juno cycle. If we go for something too complicated is going to > > >>>> take more time for approval. > > >>> > > >>> > > >>> I agree. And it not only increases chances to get at least some of > > >>> those highly demanded performance enhancements to get into Juno, > > >>> it's also "the right thing to do" (c). It's counterproductive to > > >>> put multiple vaguely related enhancements in single spec. This > > >>> would dim review focus and put us into position of getting > > >>> 'all-or-nothing'. We can't afford that. > > >>> > > >>> Let's leave one spec per enhancement. @Shihazhang, what do you > > >>> think? > > >>> > > >>> > > >>>> Also, I proposed the details of "2", trying to bring awareness > > >>>> on the topic, as I have been working with the scale lab in Red > > >>>> Hat to find and understand those issues, I have a very good > > >>>> knowledge of the problem and I believe I could make a very fast > > >>>> advance on the issue at the RPC level. > > >>> > > >>>> Given that, I'd like to work on this specific part, whether or > > >>>> not we split the specs, as it's something we believe critical > > >>>> for neutron scalability and thus, *nova parity*. > > >>> > > >>>> I will start a separate spec for "2", later on, if you find it > > >>>> ok, we keep them as separate ones, if you believe having just 1 > > >>>> spec (for 1 & 2) is going be safer for juno-* approval, then we > > >>>> can incorporate my spec in yours, but then > > >>>> "add-ipset-to-security" is not a good spec title to put all this > > >>>> together. > > >>> > > >>> > > >>>> Best regards, Miguel Ángel. > > >>> > > >>> > > >>>> On 07/02/2014 03:37 AM, shihanzhang wrote: > > >>>>> > > >>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my > > >>>>> spes, but I will also optimization the RPC from security group > > >>>>> agent to neutron server. Now the modle is > > >>>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1, > > >>>>> sg2..]', this can reduce the size of RPC respose message from > > >>>>> neutron server to security group agent. > > >>>>> > > >>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo" > > >>>>> < [email protected] > wrote: > > >>>>>> > > >>>>>> > > >>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to > > >>>>>> work into this problem, I could file an specific spec for > > >>>>>> the security group RPC optimization, a masterplan in two > > >>>>>> steps: > > >>>>>> > > >>>>>> 1) Refactor the current RPC communication for > > >>>>>> security_groups_for_devices, which could be used for full > > >>>>>> syncs, etc.. > > >>>>>> > > >>>>>> 2) Benchmark && make use of a fanout queue per security > > >>>>>> group to make sure only the hosts with instances on a > > >>>>>> certain security group get the updates as they happen. > > >>>>>> > > >>>>>> @shihanzhang do you find it reasonable? > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> ----- Original Message ----- > > >>>>>>> ----- Original Message ----- > > >>>>>>>> @Nachi: Yes that could a good improvement to factorize > > >>>>>>>> the RPC > > >>>>>>> mechanism. > > >>>>>>>> > > >>>>>>>> Another idea: What about creating a RPC topic per > > >>>>>>>> security group (quid of the > > >>>>>>> RPC topic > > >>>>>>>> scalability) on which an agent subscribes if one of its > > >>>>>>>> ports is > > >>>>>>> associated > > >>>>>>>> to the security group? > > >>>>>>>> > > >>>>>>>> Regards, Édouard. > > >>>>>>>> > > >>>>>>>> > > >>> > > >>> > > >>>>>>> Hmm, Interesting, > > >>> > > >>>>>>> @Nachi, I'm not sure I fully understood: > > >>> > > >>> > > >>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] .. > > >>>>>>> port[SG_ID1, SG_ID2], port2 , port3 > > >>> > > >>> > > >>>>>>> Probably we may need to include also the SG_IP_LIST = > > >>>>>>> [SG_IP1, SG_IP2] ... > > >>> > > >>> > > >>>>>>> and let the agent do all the combination work. > > >>> > > >>>>>>> Something like this could make sense? > > >>> > > >>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....], > > >>>>>>> SG2:{IPs:[....],RULES:[....]} } > > >>> > > >>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... } > > >>> > > >>> > > >>>>>>> @Edouard, actually I like the idea of having the agent > > >>>>>>> subscribed to security groups they have ports on... That > > >>>>>>> would remove the need to include all the security groups > > >>>>>>> information on every call... > > >>> > > >>>>>>> But would need another call to get the full information of > > >>>>>>> a set of security groups at start/resync if we don't > > >>>>>>> already have any. > > >>> > > >>> > > >>>>>>>> > > >>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang < > > >>>>>>> [email protected] > > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> hi Miguel Ángel, I am very agree with you about the > > >>>>>>>> following point: > > >>>>>>>>> * physical implementation on the hosts (ipsets, > > >>>>>>>>> nftables, ... ) > > >>>>>>>> --this can reduce the load of compute node. > > >>>>>>>>> * rpc communication mechanisms. > > >>>>>>>> -- this can reduce the load of neutron server can you > > >>>>>>>> help me to review my BP specs? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" < > > >>>>>>> [email protected] > > > >>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>> Hi it's a very interesting topic, I was getting ready > > >>>>>>>>> to raise the same concerns about our security groups > > >>>>>>>>> implementation, > > >>>>>>> shihanzhang > > >>>>>>>>> thank you for starting this topic. > > >>>>>>>>> > > >>>>>>>>> Not only at low level where (with our default security > > >>>>>>>>> group rules -allow all incoming from 'default' sg- the > > >>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the > > >>>>>>>>> "security_group_rules_for_devices" rpc call from > > >>>>>>>>> ovs-agent to neutron-server grows to message sizes of > > >>>>>>>>>> 100MB, > > >>>>>>>>> generating serious scalability issues or > > >>>>>>>>> timeouts/retries that totally break neutron service. > > >>>>>>>>> > > >>>>>>>>> (example trace of that RPC call with a few instances > > >>>>>>>>> http://www.fpaste.org/104401/14008522/ ) > > >>>>>>>>> > > >>>>>>>>> I believe that we also need to review the RPC calling > > >>>>>>>>> mechanism for the OVS agent here, there are several > > >>>>>>>>> possible approaches to > > >>>>>>> breaking > > >>>>>>>>> down (or/and CIDR compressing) the information we > > >>>>>>>>> return via this > > >>>>>>> api > > >>>>>>>>> call. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> So we have to look at two things here: > > >>>>>>>>> > > >>>>>>>>> * physical implementation on the hosts (ipsets, > > >>>>>>>>> nftables, ... ) * rpc communication mechanisms. > > >>>>>>>>> > > >>>>>>>>> Best regards, Miguel Ángel. > > >>>>>>>>> > > >>>>>>>>> ----- Mensaje original ----- > > >>>>>>>>> > > >>>>>>>>>> Do you though about nftables that will replace > > >>>>>>> {ip,ip6,arp,eb}tables? > > >>>>>>>>>> It also based on the rule set mechanism. The issue > > >>>>>>>>>> in that proposition, it's only stable since the > > >>>>>>>>>> begin > > >>>>>>> of the > > >>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros > > >>>>>>>>>> I don't list here (leverage iptables > > >>>>>>> limitation, > > >>>>>>>>>> efficient update rule, rule set, standardization of > > >>>>>>>>>> netfilter commands...). > > >>>>>>>>> > > >>>>>>>>>> Édouard. > > >>>>>>>>> > > >>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < > > >>>>>>>>>> [email protected] > wrote: > > >>>>>>>>> > > >>>>>>>>>>> we have done some tests, but have different > > >>>>>>>>>>> result: the > > >>>>>>> performance is > > >>>>>>>>>>> nearly the same for empty and 5k rules in iptable, > > >>>>>>>>>>> but huge gap between enable/disable iptable hook > > >>>>>>>>>>> on linux bridge > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < > > >>>>>>> [email protected] > > >>>>>>>>>>>> > > >>>>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>>>> Now I have not get accurate test data, but I can > > >>>>>>>>>>>> confirm the following points? > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM > > >>>>>>>>>>>> is liner, > > >>>>>>> iptable > > >>>>>>>>>>>> filter it one by one, if a VM in default > > >>>>>>>>>>>> security group and this default security group > > >>>>>>>>>>>> have many members, but ipset chain is set, the > > >>>>>>>>>>>> time > > >>>>>>> ipset > > >>>>>>>>>>>> filter one and many member is not much > > >>>>>>>>>>>> difference. > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>>> 2. when the iptable rule is very large, the > > >>>>>>>>>>>> probability of > > >>>>>>> failure > > >>>>>>>>>>>> that iptable-save save the iptable rule is very > > >>>>>>>>>>>> large. > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" < > > >>>>>>>>>>>> [email protected] > > >>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>>>>> This sounds like a good idea to handle some of > > >>>>>>>>>>>>> the > > >>>>>>> performance > > >>>>>>>>>>>>> issues until the ovs firewall can be > > >>>>>>>>>>>>> implemented down the the line. > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>>>> Do you have any performance comparisons? > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" < > > >>>>>>> [email protected] > > > >>>>>>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>>>>>> Hello all, > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>>>>>> Now in neutron, it use iptable implementing > > >>>>>>>>>>>>>> security > > >>>>>>> group, but > > >>>>>>>>>>>>>> the performance of this implementation is > > >>>>>>>>>>>>>> very poor, there > > >>>>>>> is a bug: > > >>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272 > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>to > > >>>>>>> reflect this > > >>>>>>>>>>>>>> problem. In his test, w ith default security > > >>>>>>>>>>>>>> groups(which has remote security group), > > >>>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k > > >>>>>>>>>>>>>> Iptable rules > > >>>>>>> on evry > > >>>>>>>>>>>>>> compute node, although his patch can reduce > > >>>>>>>>>>>>>> the processing time, but > > >>>>>>> it don't > > >>>>>>>>>>>>>> solve this problem fundamentally. I have > > >>>>>>>>>>>>>> commit a BP to solve this > > >>>>>>> problem: > > >>>>>>>>>>>>>> > > >>>>>>> > https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security > > >>> > > >>>>>>> > > >>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>>>>> There are other people interested in this > > >>>>>>>>>>>>>> it? > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>>>>>> _______________________________________________ > > >>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>>>>> OpenStack-dev mailing list > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>>>>> [email protected] >> > > > > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>> > > >>>>>>> > > >>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>>>> _______________________________________________ > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>>> OpenStack-dev mailing list > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>>> [email protected] >> > > > >>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>>>>>> > > >>>>>>> > > >>>> > > >>>>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>>>> _______________________________________________ > > >>>>>>>>>> > > >>>>>>>>>>> OpenStack-dev mailing list > > >>>>>>>>>> > > >>>>>>>>>>> [email protected] >> > > >>>>>>>>>>> > > >>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>>>>>> > > >>>>>>> > > >>>> > > >>>>>>>>> > > >>>>>>>>>> _______________________________________________ > > >>>>>>>>>> OpenStack-dev mailing list > > >>>>>>>>>> [email protected] >> > > >>>>>>>>>> > > >>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>>>>>> > > >>>>>>> > > >>> > > >>>>>>>>> _______________________________________________ > > >>>>>>>>> OpenStack-dev mailing list > > >>>>>>>>> [email protected] > > > >>>>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>> > > >>>>>>>>> > > >>>>>>>>> > > >>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> _______________________________________________ > > >>>>>>>> OpenStack-dev mailing list > > >>>>>>>> [email protected] > > >>>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>> > > >>>>>>>> > > >>>>>>>> > > >>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> _______________________________________________ > > >>>>>>>> OpenStack-dev mailing list > > >>>>>>>> [email protected] > > >>>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>> > > >>>>>>>> > > >>>>>>>> > > >>> > > >>> > > >>>>>>> _______________________________________________ > > >>>>>>> OpenStack-dev mailing list > > >>>>>>> [email protected] > > >>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>> > > >>>>>>> > > >>>>>> > > >>>>>> > > >>> > > >>> _______________________________________________ > > >>>>>> OpenStack-dev mailing list [email protected] > > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>>> > > >>> > > >>>>>> > > >>_______________________________________________ > > >>>>> OpenStack-dev mailing list [email protected] > > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>>>> > > >>> > > >>>>> > > >>>>> > > >>>> _______________________________________________ OpenStack-dev > > >>>> mailing list [email protected] > > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>>>>> > > >>>>>>_______________________________________________ > > >>>>>>OpenStack-dev > > >>>> > > >>mailing list > > >>>>>> [email protected] > > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>>> > > >>_______________________________________________ > > >>>>> OpenStack-dev mailing list [email protected] > > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>>>> > > >>>> > > >>>>_______________________________________________ > > >>>>OpenStack-dev > > >>>>> > > >>mailing list > > >>>> [email protected] > > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>> > > >>>> > > >>> > > >>> > > >>> > > >>> > > >>> _______________________________________________ OpenStack-dev > > >>> mailing list [email protected] > > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >>> > > >>-----BEGIN PGP SIGNATURE----- > > >>Version: GnuPG/MacGPG2 v2.0.22 (Darwin) > > >>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > >> > > >>iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR > > >>IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE > > >>WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros > > >>7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23 > > >>1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di > > >>/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc= > > >>=TxAT > > >>-----END PGP SIGNATURE----- > > >> > > >>_______________________________________________ > > >>OpenStack-dev mailing list > > >> [email protected] > > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > > > > > > _______________________________________________ > > > OpenStack-dev mailing list > > > [email protected] > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > [email protected] > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > -- > > Best wishes! > > Baohua > > > > _______________________________________________ > > OpenStack-dev mailing list > > [email protected] > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > [email protected] > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
_______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
