This is great work! Looking forward to seeing this get reviewed and merged in Juno!
Kyle On Thu, Aug 21, 2014 at 6:49 AM, Édouard Thuleau <[email protected]> wrote: > Nice job! That's awesome. > > Thanks, > Édouard. > > > On Thu, Aug 21, 2014 at 8:02 AM, Miguel Angel Ajo Pelayo > <[email protected]> wrote: >> >> Thank you shihanzhang!, >> >> I can't believe I didn't realize the ipset part spec was accepted I live >> on my own bubble... I will be reviewing and testing/helping on that part >> too during the next few days, I was too concentrated in the RPC part. >> >> >> Best regards, >> >> ----- Original Message ----- >> > hi neutroner! >> > my patch about BP: >> > >> > https://blueprints.launchpad.net/openstack/?searchtext=add-ipset-to-security >> > need install ipset in devstack, I have commit the patch: >> > https://review.openstack.org/#/c/113453/, who can help me review it, >> > thanks >> > very much! >> > >> > Best regards, >> > shihanzhang >> > >> > >> > >> > >> > At 2014-08-21 10:47:59, "Martinx - ジェームズ" <[email protected]> >> > wrote: >> > >> > >> > >> > +1 "NFTablesDriver"! >> > >> > Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example: >> > https://home.regit.org/2014/02/suricata-and-nftables/ >> > >> > Then, I'm wondering here... What benefits might come for OpenStack Nova >> > / >> > Neutron, if it comes with a NFTables driver, instead of the current >> > IPTables?! >> > >> > * E fficient Security Group design? >> > * Better FWaaS, maybe with NAT(44/66) support? >> > * Native support for IPv6, with the defamed NAT66 built-in, simpler >> > "Floating >> > IP" implementation, for both v4 and v6 networks under a single >> > implementation ( I don't like NAT66, I prefer a `routed Floating IPv6` >> > version ) ? >> > * Metadata over IPv6 still using NAT(66) ( I don't like NAT66 ), single >> > implementation? >> > * Suricata-as-a-Service?! >> > >> > It sounds pretty cool! :-) >> > >> > >> > On 20 August 2014 23:16, Baohua Yang < [email protected] > wrote: >> > >> > >> > >> > Great！ >> > We met similar problems. >> > The current mechanisms produce too many iptables rules, and it's hard to >> > debug. >> > Really look forward to seeing a more efficient security group design. >> > >> > >> > On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery < >> > [email protected] > >> > wrote: >> > >> > >> > >> > On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang < [email protected] > >> > wrote: >> > > >> > > With the deployment 'nova + neutron + openvswitch', when we bulk >> > > create >> > > about 500 VM with a default security group, the CPU usage of >> > > neutron-server >> > > and openvswitch agent is very high, especially the CPU usage of >> > > openvswitch >> > > agent will be 100%, this will cause creating VMs failed. >> > > >> > > With the method discussed in mailist: >> > > >> > > 1) ipset optimization ( https://review.openstack.org/#/c/100761/ ) >> > > >> > > 3) sg rpc optimization (with fanout) >> > > ( https://review.openstack.org/#/c/104522/ ) >> > > >> > > I have implement these two scheme in my deployment, when we again bulk >> > > create about 500 VM with a default security group, the CPU usage of >> > > openvswitch agent will reduce to 10%, even lower than 10%, so I think >> > > the >> > > iprovement of these two options are very efficient. >> > > >> > > Who can help us to review our spec? >> > > >> > This is great work! These are on my list of things to review in detail >> > soon, but given the Neutron sprint this week, I haven't had time yet. >> > I'll try to remedy that by the weekend. >> > >> > Thanks! >> > Kyle >> > >> > > Best regards, >> > > shihanzhang >> > > >> > > >> > > >> > > >> > > >> > > At 2014-07-03 10:08:21, "Ihar Hrachyshka" < [email protected] > >> > > wrote: >> > >>-----BEGIN PGP SIGNED MESSAGE----- >> > >>Hash: SHA512 >> > >> >> > >>Oh, so you have the enhancement implemented? Great! Any numbers that >> > >>shows how much we gain from that? >> > >> >> > >>/Ihar >> > >> >> > >>On 03/07/14 02:49, shihanzhang wrote: >> > >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today >> > >>> I will modify my spec, when the spec is approved, I will commit the >> > >>> codes as soon as possilbe! >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" < [email protected] > >> > >>> wrote: >> > >>>> >> > >>>> Nice Shihanzhang, >> > >>>> >> > >>>> Do you mean the ipset implementation is ready, or just the >> > >>>> spec?. >> > >>>> >> > >>>> >> > >>>> For the SG group refactor, I don't worry about who does it, or >> > >>>> who takes the credit, but I believe it's important we address >> > >>>> this bottleneck during Juno trying to match nova's scalability. >> > >>>> >> > >>>> Best regards, Miguel Ángel. >> > >>>> >> > >>>> >> > >>>> On 07/02/2014 02:50 PM, shihanzhang wrote: >> > >>>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that >> > >>>>> split the work in several specs, I have finished the work ( >> > >>>>> ipset optimization), you can do 'sg rpc optimization (without >> > >>>>> fanout)'. as the third part(sg rpc optimization (with fanout)), >> > >>>>> I think we need talk about it, because just using ipset to >> > >>>>> optimize security group agent codes does not bring the best >> > >>>>> results! >> > >>>>> >> > >>>>> Best regards, shihanzhang. >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" < [email protected] > >> > >>>>> wrote: >> > >>> On 02/07/14 10:12, Miguel Angel Ajo wrote: >> > >>> >> > >>>> Shihazhang, >> > >>> >> > >>>> I really believe we need the RPC refactor done for this cycle, >> > >>>> and given the close deadlines we have (July 10 for spec >> > >>>> submission and July 20 for spec approval). >> > >>> >> > >>>> Don't you think it's going to be better to split the work in >> > >>>> several specs? >> > >>> >> > >>>> 1) ipset optimization (you) 2) sg rpc optimization (without >> > >>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you >> > >>>> , me) >> > >>> >> > >>> >> > >>>> This way we increase the chances of having part of this for the >> > >>>> Juno cycle. If we go for something too complicated is going to >> > >>>> take more time for approval. >> > >>> >> > >>> >> > >>> I agree. And it not only increases chances to get at least some of >> > >>> those highly demanded performance enhancements to get into Juno, >> > >>> it's also "the right thing to do" (c). It's counterproductive to >> > >>> put multiple vaguely related enhancements in single spec. This >> > >>> would dim review focus and put us into position of getting >> > >>> 'all-or-nothing'. We can't afford that. >> > >>> >> > >>> Let's leave one spec per enhancement. @Shihazhang, what do you >> > >>> think? >> > >>> >> > >>> >> > >>>> Also, I proposed the details of "2", trying to bring awareness >> > >>>> on the topic, as I have been working with the scale lab in Red >> > >>>> Hat to find and understand those issues, I have a very good >> > >>>> knowledge of the problem and I believe I could make a very fast >> > >>>> advance on the issue at the RPC level. >> > >>> >> > >>>> Given that, I'd like to work on this specific part, whether or >> > >>>> not we split the specs, as it's something we believe critical >> > >>>> for neutron scalability and thus, *nova parity*. >> > >>> >> > >>>> I will start a separate spec for "2", later on, if you find it >> > >>>> ok, we keep them as separate ones, if you believe having just 1 >> > >>>> spec (for 1 & 2) is going be safer for juno-* approval, then we >> > >>>> can incorporate my spec in yours, but then >> > >>>> "add-ipset-to-security" is not a good spec title to put all this >> > >>>> together. >> > >>> >> > >>> >> > >>>> Best regards, Miguel Ángel. >> > >>> >> > >>> >> > >>>> On 07/02/2014 03:37 AM, shihanzhang wrote: >> > >>>>> >> > >>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my >> > >>>>> spes, but I will also optimization the RPC from security group >> > >>>>> agent to neutron server. Now the modle is >> > >>>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1, >> > >>>>> sg2..]', this can reduce the size of RPC respose message from >> > >>>>> neutron server to security group agent. >> > >>>>> >> > >>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo" >> > >>>>> < [email protected] > wrote: >> > >>>>>> >> > >>>>>> >> > >>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to >> > >>>>>> work into this problem, I could file an specific spec for >> > >>>>>> the security group RPC optimization, a masterplan in two >> > >>>>>> steps: >> > >>>>>> >> > >>>>>> 1) Refactor the current RPC communication for >> > >>>>>> security_groups_for_devices, which could be used for full >> > >>>>>> syncs, etc.. >> > >>>>>> >> > >>>>>> 2) Benchmark && make use of a fanout queue per security >> > >>>>>> group to make sure only the hosts with instances on a >> > >>>>>> certain security group get the updates as they happen. >> > >>>>>> >> > >>>>>> @shihanzhang do you find it reasonable? >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> ----- Original Message ----- >> > >>>>>>> ----- Original Message ----- >> > >>>>>>>> @Nachi: Yes that could a good improvement to factorize >> > >>>>>>>> the RPC >> > >>>>>>> mechanism. >> > >>>>>>>> >> > >>>>>>>> Another idea: What about creating a RPC topic per >> > >>>>>>>> security group (quid of the >> > >>>>>>> RPC topic >> > >>>>>>>> scalability) on which an agent subscribes if one of its >> > >>>>>>>> ports is >> > >>>>>>> associated >> > >>>>>>>> to the security group? >> > >>>>>>>> >> > >>>>>>>> Regards, Édouard. >> > >>>>>>>> >> > >>>>>>>> >> > >>> >> > >>> >> > >>>>>>> Hmm, Interesting, >> > >>> >> > >>>>>>> @Nachi, I'm not sure I fully understood: >> > >>> >> > >>> >> > >>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] .. >> > >>>>>>> port[SG_ID1, SG_ID2], port2 , port3 >> > >>> >> > >>> >> > >>>>>>> Probably we may need to include also the SG_IP_LIST = >> > >>>>>>> [SG_IP1, SG_IP2] ... >> > >>> >> > >>> >> > >>>>>>> and let the agent do all the combination work. >> > >>> >> > >>>>>>> Something like this could make sense? >> > >>> >> > >>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....], >> > >>>>>>> SG2:{IPs:[....],RULES:[....]} } >> > >>> >> > >>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... } >> > >>> >> > >>> >> > >>>>>>> @Edouard, actually I like the idea of having the agent >> > >>>>>>> subscribed to security groups they have ports on... That >> > >>>>>>> would remove the need to include all the security groups >> > >>>>>>> information on every call... >> > >>> >> > >>>>>>> But would need another call to get the full information of >> > >>>>>>> a set of security groups at start/resync if we don't >> > >>>>>>> already have any. >> > >>> >> > >>> >> > >>>>>>>> >> > >>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang < >> > >>>>>>> [email protected] > >> > >>>>>>>> wrote: >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> hi Miguel Ángel, I am very agree with you about the >> > >>>>>>>> following point: >> > >>>>>>>>> * physical implementation on the hosts (ipsets, >> > >>>>>>>>> nftables, ... ) >> > >>>>>>>> --this can reduce the load of compute node. >> > >>>>>>>>> * rpc communication mechanisms. >> > >>>>>>>> -- this can reduce the load of neutron server can you >> > >>>>>>>> help me to review my BP specs? >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" < >> > >>>>>>> [email protected] > >> > >>>>>>>> wrote: >> > >>>>>>>>> >> > >>>>>>>>> Hi it's a very interesting topic, I was getting ready >> > >>>>>>>>> to raise the same concerns about our security groups >> > >>>>>>>>> implementation, >> > >>>>>>> shihanzhang >> > >>>>>>>>> thank you for starting this topic. >> > >>>>>>>>> >> > >>>>>>>>> Not only at low level where (with our default security >> > >>>>>>>>> group rules -allow all incoming from 'default' sg- the >> > >>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the >> > >>>>>>>>> "security_group_rules_for_devices" rpc call from >> > >>>>>>>>> ovs-agent to neutron-server grows to message sizes of >> > >>>>>>>>>> 100MB, >> > >>>>>>>>> generating serious scalability issues or >> > >>>>>>>>> timeouts/retries that totally break neutron service. >> > >>>>>>>>> >> > >>>>>>>>> (example trace of that RPC call with a few instances >> > >>>>>>>>> http://www.fpaste.org/104401/14008522/ ) >> > >>>>>>>>> >> > >>>>>>>>> I believe that we also need to review the RPC calling >> > >>>>>>>>> mechanism for the OVS agent here, there are several >> > >>>>>>>>> possible approaches to >> > >>>>>>> breaking >> > >>>>>>>>> down (or/and CIDR compressing) the information we >> > >>>>>>>>> return via this >> > >>>>>>> api >> > >>>>>>>>> call. >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> So we have to look at two things here: >> > >>>>>>>>> >> > >>>>>>>>> * physical implementation on the hosts (ipsets, >> > >>>>>>>>> nftables, ... ) * rpc communication mechanisms. >> > >>>>>>>>> >> > >>>>>>>>> Best regards, Miguel Ángel. >> > >>>>>>>>> >> > >>>>>>>>> ----- Mensaje original ----- >> > >>>>>>>>> >> > >>>>>>>>>> Do you though about nftables that will replace >> > >>>>>>> {ip,ip6,arp,eb}tables? >> > >>>>>>>>>> It also based on the rule set mechanism. The issue >> > >>>>>>>>>> in that proposition, it's only stable since the >> > >>>>>>>>>> begin >> > >>>>>>> of the >> > >>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros >> > >>>>>>>>>> I don't list here (leverage iptables >> > >>>>>>> limitation, >> > >>>>>>>>>> efficient update rule, rule set, standardization of >> > >>>>>>>>>> netfilter commands...). >> > >>>>>>>>> >> > >>>>>>>>>> Édouard. >> > >>>>>>>>> >> > >>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < >> > >>>>>>>>>> [email protected] > wrote: >> > >>>>>>>>> >> > >>>>>>>>>>> we have done some tests, but have different >> > >>>>>>>>>>> result: the >> > >>>>>>> performance is >> > >>>>>>>>>>> nearly the same for empty and 5k rules in iptable, >> > >>>>>>>>>>> but huge gap between enable/disable iptable hook >> > >>>>>>>>>>> on linux bridge >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < >> > >>>>>>> [email protected] >> > >>>>>>>>>>>> >> > >>>>>>>>>>> wrote: >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>>>>> Now I have not get accurate test data, but I can >> > >>>>>>>>>>>> confirm the following points? >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM >> > >>>>>>>>>>>> is liner, >> > >>>>>>> iptable >> > >>>>>>>>>>>> filter it one by one, if a VM in default >> > >>>>>>>>>>>> security group and this default security group >> > >>>>>>>>>>>> have many members, but ipset chain is set, the >> > >>>>>>>>>>>> time >> > >>>>>>> ipset >> > >>>>>>>>>>>> filter one and many member is not much >> > >>>>>>>>>>>> difference. >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>>>> 2. when the iptable rule is very large, the >> > >>>>>>>>>>>> probability of >> > >>>>>>> failure >> > >>>>>>>>>>>> that iptable-save save the iptable rule is very >> > >>>>>>>>>>>> large. >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" < >> > >>>>>>>>>>>> [email protected] >> > >>>>>>>> wrote: >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>>>>>> This sounds like a good idea to handle some of >> > >>>>>>>>>>>>> the >> > >>>>>>> performance >> > >>>>>>>>>>>>> issues until the ovs firewall can be >> > >>>>>>>>>>>>> implemented down the the line. >> > >>>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>>>>> Do you have any performance comparisons? >> > >>>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" < >> > >>>>>>> [email protected] > >> > >>>>>>>>>>>>> wrote: >> > >>>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>>>>>>> Hello all, >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>>>>>>> Now in neutron, it use iptable implementing >> > >>>>>>>>>>>>>> security >> > >>>>>>> group, but >> > >>>>>>>>>>>>>> the performance of this implementation is >> > >>>>>>>>>>>>>> very poor, there >> > >>>>>>> is a bug: >> > >>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272 >> > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > >>to >> > >>>>>>> reflect this >> > >>>>>>>>>>>>>> problem. In his test, w ith default security >> > >>>>>>>>>>>>>> groups(which has remote security group), >> > >>>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k >> > >>>>>>>>>>>>>> Iptable rules >> > >>>>>>> on evry >> > >>>>>>>>>>>>>> compute node, although his patch can reduce >> > >>>>>>>>>>>>>> the processing time, but >> > >>>>>>> it don't >> > >>>>>>>>>>>>>> solve this problem fundamentally. I have >> > >>>>>>>>>>>>>> commit a BP to solve this >> > >>>>>>> problem: >> > >>>>>>>>>>>>>> >> > >>>>>>> >> > >>>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security >> > >>> >> > >>>>>>> >> > >>> >> > >>>>>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>>>>>> There are other people interested in this >> > >>>>>>>>>>>>>> it? >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>>>>>>> _______________________________________________ >> > >>> >> > >>>>>>>>>>>>>> >> > >>>>>>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>>>>>> OpenStack-dev mailing list >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>>>>>> [email protected] >> > > > >> > >>>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>>>>>> >> > >>>>>>> >> > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>> >> > >>>>>>> >> > >>> >> > >>>>>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>>>>> _______________________________________________ >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>>>> OpenStack-dev mailing list >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>>>> [email protected] >> > >> > >>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>> >> > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>>>>>> >> > >>>>>>> >> > >>>> >> > >>>>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>>>> _______________________________________________ >> > >>>>>>>>>> >> > >>>>>>>>>>> OpenStack-dev mailing list >> > >>>>>>>>>> >> > >>>>>>>>>>> [email protected] >> >> > >>>>>>>>>>> >> > >>>>>>> >> > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>>>>>> >> > >>>>>>> >> > >>>> >> > >>>>>>>>> >> > >>>>>>>>>> _______________________________________________ >> > >>>>>>>>>> OpenStack-dev mailing list >> > >>>>>>>>>> [email protected] >> >> > >>>>>>>>>> >> > >>>>>>> >> > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>>>>>> >> > >>>>>>> >> > >>> >> > >>>>>>>>> _______________________________________________ >> > >>>>>>>>> OpenStack-dev mailing list >> > >>>>>>>>> [email protected] > >> > >>>>>>>>> >> > >>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>> >> > >>>>>>>>> >> > >>>>>>>>> >> > >>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> _______________________________________________ >> > >>>>>>>> OpenStack-dev mailing list >> > >>>>>>>> [email protected] >> > >>>>>>>> >> > >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>> >> > >>>>>>>> >> > >>>>>>>> >> > >>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> _______________________________________________ >> > >>>>>>>> OpenStack-dev mailing list >> > >>>>>>>> [email protected] >> > >>>>>>>> >> > >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>> >> > >>>>>>>> >> > >>>>>>>> >> > >>> >> > >>> >> > >>>>>>> _______________________________________________ >> > >>>>>>> OpenStack-dev mailing list >> > >>>>>>> [email protected] >> > >>>>>>> >> > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>> >> > >>>>>>> >> > >>>>>> >> > >>>>>> >> > >>> >> > >>> _______________________________________________ >> > >>>>>> OpenStack-dev mailing list [email protected] >> > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>>> >> > >>> >> > >>>>>> >> > >>_______________________________________________ >> > >>>>> OpenStack-dev mailing list [email protected] >> > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>>>> >> > >>> >> > >>>>> >> > >>>>> >> > >>>> _______________________________________________ OpenStack-dev >> > >>>> mailing list [email protected] >> > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>>>>> >> > >>>>>>_______________________________________________ >> > >>>>>>OpenStack-dev >> > >>>> >> > >>mailing list >> > >>>>>> [email protected] >> > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>>> >> > >>_______________________________________________ >> > >>>>> OpenStack-dev mailing list [email protected] >> > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>>>> >> > >>>> >> > >>>>_______________________________________________ >> > >>>>OpenStack-dev >> > >>>>> >> > >>mailing list >> > >>>> [email protected] >> > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>> >> > >>>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> _______________________________________________ OpenStack-dev >> > >>> mailing list [email protected] >> > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >>> >> > >>-----BEGIN PGP SIGNATURE----- >> > >>Version: GnuPG/MacGPG2 v2.0.22 (Darwin) >> > >>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >> > >> >> > >>iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR >> > >>IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE >> > >>WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros >> > >>7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23 >> > >>1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di >> > >>/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc= >> > >>=TxAT >> > >>-----END PGP SIGNATURE----- >> > >> >> > >>_______________________________________________ >> > >>OpenStack-dev mailing list >> > >> [email protected] >> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > >> > > >> > > >> > > >> > > _______________________________________________ >> > > OpenStack-dev mailing list >> > > [email protected] >> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > >> > >> > _______________________________________________ >> > OpenStack-dev mailing list >> > [email protected] >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > >> > >> > -- >> > Best wishes! >> > Baohua >> > >> > _______________________________________________ >> > OpenStack-dev mailing list >> > [email protected] >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > >> > >> > >> > >> > _______________________________________________ >> > OpenStack-dev mailing list >> > [email protected] >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> _______________________________________________ >> OpenStack-dev mailing list >> [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
