On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang <ayshihanzh...@126.com> wrote: > > With the deployment 'nova + neutron + openvswitch', when we bulk create > about 500 VM with a default security group, the CPU usage of neutron-server > and openvswitch agent is very high, especially the CPU usage of openvswitch > agent will be 100%, this will cause creating VMs failed. > > With the method discussed in mailist: > > 1) ipset optimization (https://review.openstack.org/#/c/100761/) > > 3) sg rpc optimization (with fanout) > (https://review.openstack.org/#/c/104522/) > > I have implement these two scheme in my deployment, when we again bulk > create about 500 VM with a default security group, the CPU usage of > openvswitch agent will reduce to 10%, even lower than 10%, so I think the > iprovement of these two options are very efficient. > > Who can help us to review our spec? > This is great work! These are on my list of things to review in detail soon, but given the Neutron sprint this week, I haven't had time yet. I'll try to remedy that by the weekend.
Thanks! Kyle > Best regards, > shihanzhang > > > > > > At 2014-07-03 10:08:21, "Ihar Hrachyshka" <ihrac...@redhat.com> wrote: >>-----BEGIN PGP SIGNED MESSAGE----- >>Hash: SHA512 >> >>Oh, so you have the enhancement implemented? Great! Any numbers that >>shows how much we gain from that? >> >>/Ihar >> >>On 03/07/14 02:49, shihanzhang wrote: >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today >>> I will modify my spec, when the spec is approved, I will commit the >>> codes as soon as possilbe! >>> >>> >>> >>> >>> >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" <majop...@redhat.com> >>> wrote: >>>> >>>> Nice Shihanzhang, >>>> >>>> Do you mean the ipset implementation is ready, or just the >>>> spec?. >>>> >>>> >>>> For the SG group refactor, I don't worry about who does it, or >>>> who takes the credit, but I believe it's important we address >>>> this bottleneck during Juno trying to match nova's scalability. >>>> >>>> Best regards, Miguel Ángel. >>>> >>>> >>>> On 07/02/2014 02:50 PM, shihanzhang wrote: >>>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that >>>>> split the work in several specs, I have finished the work ( >>>>> ipset optimization), you can do 'sg rpc optimization (without >>>>> fanout)'. as the third part(sg rpc optimization (with fanout)), >>>>> I think we need talk about it, because just using ipset to >>>>> optimize security group agent codes does not bring the best >>>>> results! >>>>> >>>>> Best regards, shihanzhang. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" <ihrac...@redhat.com> >>>>> wrote: >>> On 02/07/14 10:12, Miguel Angel Ajo wrote: >>> >>>> Shihazhang, >>> >>>> I really believe we need the RPC refactor done for this cycle, >>>> and given the close deadlines we have (July 10 for spec >>>> submission and July 20 for spec approval). >>> >>>> Don't you think it's going to be better to split the work in >>>> several specs? >>> >>>> 1) ipset optimization (you) 2) sg rpc optimization (without >>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you >>>> , me) >>> >>> >>>> This way we increase the chances of having part of this for the >>>> Juno cycle. If we go for something too complicated is going to >>>> take more time for approval. >>> >>> >>> I agree. And it not only increases chances to get at least some of >>> those highly demanded performance enhancements to get into Juno, >>> it's also "the right thing to do" (c). It's counterproductive to >>> put multiple vaguely related enhancements in single spec. This >>> would dim review focus and put us into position of getting >>> 'all-or-nothing'. We can't afford that. >>> >>> Let's leave one spec per enhancement. @Shihazhang, what do you >>> think? >>> >>> >>>> Also, I proposed the details of "2", trying to bring awareness >>>> on the topic, as I have been working with the scale lab in Red >>>> Hat to find and understand those issues, I have a very good >>>> knowledge of the problem and I believe I could make a very fast >>>> advance on the issue at the RPC level. >>> >>>> Given that, I'd like to work on this specific part, whether or >>>> not we split the specs, as it's something we believe critical >>>> for neutron scalability and thus, *nova parity*. >>> >>>> I will start a separate spec for "2", later on, if you find it >>>> ok, we keep them as separate ones, if you believe having just 1 >>>> spec (for 1 & 2) is going be safer for juno-* approval, then we >>>> can incorporate my spec in yours, but then >>>> "add-ipset-to-security" is not a good spec title to put all this >>>> together. >>> >>> >>>> Best regards, Miguel Ángel. >>> >>> >>>> On 07/02/2014 03:37 AM, shihanzhang wrote: >>>>> >>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my >>>>> spes, but I will also optimization the RPC from security group >>>>> agent to neutron server. Now the modle is >>>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1, >>>>> sg2..]', this can reduce the size of RPC respose message from >>>>> neutron server to security group agent. >>>>> >>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo" >>>>> <mangel...@redhat.com> wrote: >>>>>> >>>>>> >>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to >>>>>> work into this problem, I could file an specific spec for >>>>>> the security group RPC optimization, a masterplan in two >>>>>> steps: >>>>>> >>>>>> 1) Refactor the current RPC communication for >>>>>> security_groups_for_devices, which could be used for full >>>>>> syncs, etc.. >>>>>> >>>>>> 2) Benchmark && make use of a fanout queue per security >>>>>> group to make sure only the hosts with instances on a >>>>>> certain security group get the updates as they happen. >>>>>> >>>>>> @shihanzhang do you find it reasonable? >>>>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> ----- Original Message ----- >>>>>>>> @Nachi: Yes that could a good improvement to factorize >>>>>>>> the RPC >>>>>>> mechanism. >>>>>>>> >>>>>>>> Another idea: What about creating a RPC topic per >>>>>>>> security group (quid of the >>>>>>> RPC topic >>>>>>>> scalability) on which an agent subscribes if one of its >>>>>>>> ports is >>>>>>> associated >>>>>>>> to the security group? >>>>>>>> >>>>>>>> Regards, Édouard. >>>>>>>> >>>>>>>> >>> >>> >>>>>>> Hmm, Interesting, >>> >>>>>>> @Nachi, I'm not sure I fully understood: >>> >>> >>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] .. >>>>>>> port[SG_ID1, SG_ID2], port2 , port3 >>> >>> >>>>>>> Probably we may need to include also the SG_IP_LIST = >>>>>>> [SG_IP1, SG_IP2] ... >>> >>> >>>>>>> and let the agent do all the combination work. >>> >>>>>>> Something like this could make sense? >>> >>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....], >>>>>>> SG2:{IPs:[....],RULES:[....]} } >>> >>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... } >>> >>> >>>>>>> @Edouard, actually I like the idea of having the agent >>>>>>> subscribed to security groups they have ports on... That >>>>>>> would remove the need to include all the security groups >>>>>>> information on every call... >>> >>>>>>> But would need another call to get the full information of >>>>>>> a set of security groups at start/resync if we don't >>>>>>> already have any. >>> >>> >>>>>>>> >>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang < >>>>>>> ayshihanzh...@126.com > >>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> hi Miguel Ángel, I am very agree with you about the >>>>>>>> following point: >>>>>>>>> * physical implementation on the hosts (ipsets, >>>>>>>>> nftables, ... ) >>>>>>>> --this can reduce the load of compute node. >>>>>>>>> * rpc communication mechanisms. >>>>>>>> -- this can reduce the load of neutron server can you >>>>>>>> help me to review my BP specs? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" < >>>>>>> mangel...@redhat.com > >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi it's a very interesting topic, I was getting ready >>>>>>>>> to raise the same concerns about our security groups >>>>>>>>> implementation, >>>>>>> shihanzhang >>>>>>>>> thank you for starting this topic. >>>>>>>>> >>>>>>>>> Not only at low level where (with our default security >>>>>>>>> group rules -allow all incoming from 'default' sg- the >>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the >>>>>>>>> "security_group_rules_for_devices" rpc call from >>>>>>>>> ovs-agent to neutron-server grows to message sizes of >>>>>>>>>> 100MB, >>>>>>>>> generating serious scalability issues or >>>>>>>>> timeouts/retries that totally break neutron service. >>>>>>>>> >>>>>>>>> (example trace of that RPC call with a few instances >>>>>>>>> http://www.fpaste.org/104401/14008522/ ) >>>>>>>>> >>>>>>>>> I believe that we also need to review the RPC calling >>>>>>>>> mechanism for the OVS agent here, there are several >>>>>>>>> possible approaches to >>>>>>> breaking >>>>>>>>> down (or/and CIDR compressing) the information we >>>>>>>>> return via this >>>>>>> api >>>>>>>>> call. >>>>>>>>> >>>>>>>>> >>>>>>>>> So we have to look at two things here: >>>>>>>>> >>>>>>>>> * physical implementation on the hosts (ipsets, >>>>>>>>> nftables, ... ) * rpc communication mechanisms. >>>>>>>>> >>>>>>>>> Best regards, Miguel Ángel. >>>>>>>>> >>>>>>>>> ----- Mensaje original ----- >>>>>>>>> >>>>>>>>>> Do you though about nftables that will replace >>>>>>> {ip,ip6,arp,eb}tables? >>>>>>>>>> It also based on the rule set mechanism. The issue >>>>>>>>>> in that proposition, it's only stable since the >>>>>>>>>> begin >>>>>>> of the >>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros >>>>>>>>>> I don't list here (leverage iptables >>>>>>> limitation, >>>>>>>>>> efficient update rule, rule set, standardization of >>>>>>>>>> netfilter commands...). >>>>>>>>> >>>>>>>>>> Édouard. >>>>>>>>> >>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < >>>>>>>>>> henry4...@gmail.com > wrote: >>>>>>>>> >>>>>>>>>>> we have done some tests, but have different >>>>>>>>>>> result: the >>>>>>> performance is >>>>>>>>>>> nearly the same for empty and 5k rules in iptable, >>>>>>>>>>> but huge gap between enable/disable iptable hook >>>>>>>>>>> on linux bridge >>>>>>>>>> >>>>>>>>> >>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < >>>>>>> ayshihanzh...@126.com >>>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> Now I have not get accurate test data, but I can >>>>>>>>>>>> confirm the following points? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM >>>>>>>>>>>> is liner, >>>>>>> iptable >>>>>>>>>>>> filter it one by one, if a VM in default >>>>>>>>>>>> security group and this default security group >>>>>>>>>>>> have many members, but ipset chain is set, the >>>>>>>>>>>> time >>>>>>> ipset >>>>>>>>>>>> filter one and many member is not much >>>>>>>>>>>> difference. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> 2. when the iptable rule is very large, the >>>>>>>>>>>> probability of >>>>>>> failure >>>>>>>>>>>> that iptable-save save the iptable rule is very >>>>>>>>>>>> large. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" < >>>>>>>>>>>> blak...@gmail.com >>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>> This sounds like a good idea to handle some of >>>>>>>>>>>>> the >>>>>>> performance >>>>>>>>>>>>> issues until the ovs firewall can be >>>>>>>>>>>>> implemented down the the line. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> Do you have any performance comparisons? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" < >>>>>>> ayshihanzh...@126.com > >>>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> Hello all, >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> Now in neutron, it use iptable implementing >>>>>>>>>>>>>> security >>>>>>> group, but >>>>>>>>>>>>>> the performance of this implementation is >>>>>>>>>>>>>> very poor, there >>>>>>> is a bug: >>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>to >>>>>>> reflect this >>>>>>>>>>>>>> problem. In his test, w ith default security >>>>>>>>>>>>>> groups(which has remote security group), >>>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k >>>>>>>>>>>>>> Iptable rules >>>>>>> on evry >>>>>>>>>>>>>> compute node, although his patch can reduce >>>>>>>>>>>>>> the processing time, but >>>>>>> it don't >>>>>>>>>>>>>> solve this problem fundamentally. I have >>>>>>>>>>>>>> commit a BP to solve this >>>>>>> problem: >>>>>>>>>>>>>> >>>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security >>> >>>>>>> >>> >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>> There are other people interested in this >>>>>>>>>>>>>> it? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>> OpenStack-dev mailing list >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>> OpenStack-dev@lists.openstack.org >> > > > >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>>>>>> >>> >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> OpenStack-dev mailing list >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> OpenStack-dev@lists.openstack.org >> > >>>>>>>>>> >>>>>>>>>>>> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>> >>>>>>> >>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>> >>>>>>>>>>> OpenStack-dev mailing list >>>>>>>>>> >>>>>>>>>>> OpenStack-dev@lists.openstack.org >> >>>>>>>>>>> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>> >>>>>>> >>>> >>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> OpenStack-dev mailing list >>>>>>>>>> OpenStack-dev@lists.openstack.org >> >>>>>>>>>> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>> >>>>>>> >>> >>>>>>>>> _______________________________________________ >>>>>>>>> OpenStack-dev mailing list >>>>>>>>> OpenStack-dev@lists.openstack.org > >>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>>>>>>>> >>>>>>>>> >>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> OpenStack-dev mailing list >>>>>>>> OpenStack-dev@lists.openstack.org >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>>>>>>> >>>>>>>> >>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> OpenStack-dev mailing list >>>>>>>> OpenStack-dev@lists.openstack.org >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>>>>>>> >>>>>>>> >>> >>> >>>>>>> _______________________________________________ >>>>>>> OpenStack-dev mailing list >>>>>>> OpenStack-dev@lists.openstack.org >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>>>>>> >>>>>> >>>>>> >>> >>> _______________________________________________ >>>>>> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> >>> >>>>>> >>_______________________________________________ >>>>> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>> >>>>> >>>>> >>>> _______________________________________________ OpenStack-dev >>>> mailing list OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>>_______________________________________________ >>>>>>OpenStack-dev >>>> >>mailing list >>>>>> OpenStack-dev@lists.openstack.org >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> >>_______________________________________________ >>>>> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>> >>>>_______________________________________________ >>>>OpenStack-dev >>>>> >>mailing list >>>> OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>>> >>> >>> >>> >>> >>> _______________________________________________ OpenStack-dev >>> mailing list OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>-----BEGIN PGP SIGNATURE----- >>Version: GnuPG/MacGPG2 v2.0.22 (Darwin) >>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >> >>iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR >>IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE >>WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros >>7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23 >>1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di >>/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc= >>=TxAT >>-----END PGP SIGNATURE----- >> >>_______________________________________________ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev