Wow Shihanzhang, truly awesome!, Is the current implementation for https://review.openstack.org/#/c/104522/ also ready?, could you make it available?.
Good work!, ----- Original Message ----- > > With the deployment 'nova + neutron + openvswitch', when we bulk create about > 500 VM with a default security group, the CPU usage of neutron-server and > openvswitch agent is very high, especially the CPU usage of openvswitch > agent will be 100%, this will cause creating VMs failed. > > With the method discussed in mailist: > 1) ipset optimization (https://review.openstack.org/#/c/100761/) > 3) sg rpc optimization (with fanout) ( > https://review.openstack.org/#/c/104522/) > I have implement these two scheme in my deployment, when we again bulk > create about 500 VM with a default security group, the CPU usage of > openvswitch agent will reduce to 10%, e ven lower than 10%, so I think the i > provement of these two options are very efficient. > Who can help us to review our spec? > Best regards, > shihanzhang > > > > > > At 2014-07-03 10:08:21, "Ihar Hrachyshka" <[email protected]> wrote: > >-----BEGIN PGP SIGNED MESSAGE----- > >Hash: SHA512 > > > >Oh, so you have the enhancement implemented? Great! Any numbers that > >shows how much we gain from that? > > > >/Ihar > > > >On 03/07/14 02:49, shihanzhang wrote: > >> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today > >> I will modify my spec, when the spec is approved, I will commit the > >> codes as soon as possilbe! > >> > >> > >> > >> > >> > >> At 2014-07-02 10:12:34, "Miguel Angel Ajo" <[email protected]> > >> wrote: > >>> > >>> Nice Shihanzhang, > >>> > >>> Do you mean the ipset implementation is ready, or just the > >>> spec?. > >>> > >>> > >>> For the SG group refactor, I don't worry about who does it, or > >>> who takes the credit, but I believe it's important we address > >>> this bottleneck during Juno trying to match nova's scalability. > >>> > >>> Best regards, Miguel Ángel. > >>> > >>> > >>> On 07/02/2014 02:50 PM, shihanzhang wrote: > >>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that > >>>> split the work in several specs, I have finished the work ( > >>>> ipset optimization), you can do 'sg rpc optimization (without > >>>> fanout)'. as the third part(sg rpc optimization (with fanout)), > >>>> I think we need talk about it, because just using ipset to > >>>> optimize security group agent codes does not bring the best > >>>> results! > >>>> > >>>> Best regards, shihanzhang. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" <[email protected]> > >>>> wrote: > >> On 02/07/14 10:12, Miguel Angel Ajo wrote: > >> > >>> Shihazhang, > >> > >>> I really believe we need the RPC refactor done for this cycle, > >>> and given the close deadlines we have (July 10 for spec > >>> submission and July 20 for spec approval). > >> > >>> Don't you think it's going to be better to split the work in > >>> several specs? > >> > >>> 1) ipset optimization (you) 2) sg rpc optimization (without > >>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you > >>> , me) > >> > >> > >>> This way we increase the chances of having part of this for the > >>> Juno cycle. If we go for something too complicated is going to > >>> take more time for approval. > >> > >> > >> I agree. And it not only increases chances to get at least some of > >> those highly demanded performance enhancements to get into Juno, > >> it's also "the right thing to do" (c). It's counterproductive to > >> put multiple vaguely related enhancements in single spec. This > >> would dim review focus and put us into position of getting > >> 'all-or-nothing'. We can't afford that. > >> > >> Let's leave one spec per enhancement. @Shihazhang, what do you > >> think? > >> > >> > >>> Also, I proposed the details of "2", trying to bring awareness > >>> on the topic, as I have been working with the scale lab in Red > >>> Hat to find and understand those issues, I have a very good > >>> knowledge of the problem and I believe I could make a very fast > >>> advance on the issue at the RPC level. > >> > >>> Given that, I'd like to work on this specific part, whether or > >>> not we split the specs, as it's something we believe critical > >>> for neutron scalability and thus, *nova parity*. > >> > >>> I will start a separate spec for "2", later on, if you find it > >>> ok, we keep them as separate ones, if you believe having just 1 > >>> spec (for 1 & 2) is going be safer for juno-* approval, then we > >>> can incorporate my spec in yours, but then > >>> "add-ipset-to-security" is not a good spec title to put all this > >>> together. > >> > >> > >>> Best regards, Miguel Ángel. > >> > >> > >>> On 07/02/2014 03:37 AM, shihanzhang wrote: > >>>> > >>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my > >>>> spes, but I will also optimization the RPC from security group > >>>> agent to neutron server. Now the modle is > >>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1, > >>>> sg2..]', this can reduce the size of RPC respose message from > >>>> neutron server to security group agent. > >>>> > >>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo" > >>>> <[email protected]> wrote: > >>>>> > >>>>> > >>>>> Ok, I was talking with Édouard @ IRC, and as I have time to > >>>>> work into this problem, I could file an specific spec for > >>>>> the security group RPC optimization, a masterplan in two > >>>>> steps: > >>>>> > >>>>> 1) Refactor the current RPC communication for > >>>>> security_groups_for_devices, which could be used for full > >>>>> syncs, etc.. > >>>>> > >>>>> 2) Benchmark && make use of a fanout queue per security > >>>>> group to make sure only the hosts with instances on a > >>>>> certain security group get the updates as they happen. > >>>>> > >>>>> @shihanzhang do you find it reasonable? > >>>>> > >>>>> > >>>>> > >>>>> ----- Original Message ----- > >>>>>> ----- Original Message ----- > >>>>>>> @Nachi: Yes that could a good improvement to factorize > >>>>>>> the RPC > >>>>>> mechanism. > >>>>>>> > >>>>>>> Another idea: What about creating a RPC topic per > >>>>>>> security group (quid of the > >>>>>> RPC topic > >>>>>>> scalability) on which an agent subscribes if one of its > >>>>>>> ports is > >>>>>> associated > >>>>>>> to the security group? > >>>>>>> > >>>>>>> Regards, Édouard. > >>>>>>> > >>>>>>> > >> > >> > >>>>>> Hmm, Interesting, > >> > >>>>>> @Nachi, I'm not sure I fully understood: > >> > >> > >>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] .. > >>>>>> port[SG_ID1, SG_ID2], port2 , port3 > >> > >> > >>>>>> Probably we may need to include also the SG_IP_LIST = > >>>>>> [SG_IP1, SG_IP2] ... > >> > >> > >>>>>> and let the agent do all the combination work. > >> > >>>>>> Something like this could make sense? > >> > >>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....], > >>>>>> SG2:{IPs:[....],RULES:[....]} } > >> > >>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... } > >> > >> > >>>>>> @Edouard, actually I like the idea of having the agent > >>>>>> subscribed to security groups they have ports on... That > >>>>>> would remove the need to include all the security groups > >>>>>> information on every call... > >> > >>>>>> But would need another call to get the full information of > >>>>>> a set of security groups at start/resync if we don't > >>>>>> already have any. > >> > >> > >>>>>>> > >>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang < > >>>>>> [email protected] > > >>>>>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> hi Miguel Ángel, I am very agree with you about the > >>>>>>> following point: > >>>>>>>> * physical implementation on the hosts (ipsets, > >>>>>>>> nftables, ... ) > >>>>>>> --this can reduce the load of compute node. > >>>>>>>> * rpc communication mechanisms. > >>>>>>> -- this can reduce the load of neutron server can you > >>>>>>> help me to review my BP specs? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" < > >>>>>> [email protected] > > >>>>>>> wrote: > >>>>>>>> > >>>>>>>> Hi it's a very interesting topic, I was getting ready > >>>>>>>> to raise the same concerns about our security groups > >>>>>>>> implementation, > >>>>>> shihanzhang > >>>>>>>> thank you for starting this topic. > >>>>>>>> > >>>>>>>> Not only at low level where (with our default security > >>>>>>>> group rules -allow all incoming from 'default' sg- the > >>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the > >>>>>>>> "security_group_rules_for_devices" rpc call from > >>>>>>>> ovs-agent to neutron-server grows to message sizes of > >>>>>>>>> 100MB, > >>>>>>>> generating serious scalability issues or > >>>>>>>> timeouts/retries that totally break neutron service. > >>>>>>>> > >>>>>>>> (example trace of that RPC call with a few instances > >>>>>>>> http://www.fpaste.org/104401/14008522/ ) > >>>>>>>> > >>>>>>>> I believe that we also need to review the RPC calling > >>>>>>>> mechanism for the OVS agent here, there are several > >>>>>>>> possible approaches to > >>>>>> breaking > >>>>>>>> down (or/and CIDR compressing) the information we > >>>>>>>> return via this > >>>>>> api > >>>>>>>> call. > >>>>>>>> > >>>>>>>> > >>>>>>>> So we have to look at two things here: > >>>>>>>> > >>>>>>>> * physical implementation on the hosts (ipsets, > >>>>>>>> nftables, ... ) * rpc communication mechanisms. > >>>>>>>> > >>>>>>>> Best regards, Miguel Ángel. > >>>>>>>> > >>>>>>>> ----- Mensaje original ----- > >>>>>>>> > >>>>>>>>> Do you though about nftables that will replace > >>>>>> {ip,ip6,arp,eb}tables? > >>>>>>>>> It also based on the rule set mechanism. The issue > >>>>>>>>> in that proposition, it's only stable since the > >>>>>>>>> begin > >>>>>> of the > >>>>>>>>> year and on Linux kernel 3.13. But there lot of pros > >>>>>>>>> I don't list here (leverage iptables > >>>>>> limitation, > >>>>>>>>> efficient update rule, rule set, standardization of > >>>>>>>>> netfilter commands...). > >>>>>>>> > >>>>>>>>> Édouard. > >>>>>>>> > >>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < > >>>>>>>>> [email protected] > wrote: > >>>>>>>> > >>>>>>>>>> we have done some tests, but have different > >>>>>>>>>> result: the > >>>>>> performance is > >>>>>>>>>> nearly the same for empty and 5k rules in iptable, > >>>>>>>>>> but huge gap between enable/disable iptable hook > >>>>>>>>>> on linux bridge > >>>>>>>>> > >>>>>>>> > >>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < > >>>>>> [email protected] > >>>>>>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>> > >>>>>>>>>>> Now I have not get accurate test data, but I can > >>>>>>>>>>> confirm the following points? > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>> 1. In compute node, the iptable's chain of a VM > >>>>>>>>>>> is liner, > >>>>>> iptable > >>>>>>>>>>> filter it one by one, if a VM in default > >>>>>>>>>>> security group and this default security group > >>>>>>>>>>> have many members, but ipset chain is set, the > >>>>>>>>>>> time > >>>>>> ipset > >>>>>>>>>>> filter one and many member is not much > >>>>>>>>>>> difference. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>> 2. when the iptable rule is very large, the > >>>>>>>>>>> probability of > >>>>>> failure > >>>>>>>>>>> that iptable-save save the iptable rule is very > >>>>>>>>>>> large. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" < > >>>>>>>>>>> [email protected] > >>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> This sounds like a good idea to handle some of > >>>>>>>>>>>> the > >>>>>> performance > >>>>>>>>>>>> issues until the ovs firewall can be > >>>>>>>>>>>> implemented down the the line. > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> Do you have any performance comparisons? > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" < > >>>>>> [email protected] > > >>>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>> Hello all, > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>> Now in neutron, it use iptable implementing > >>>>>>>>>>>>> security > >>>>>> group, but > >>>>>>>>>>>>> the performance of this implementation is > >>>>>>>>>>>>> very poor, there > >>>>>> is a bug: > >>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272 > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >to > >>>>>> reflect this > >>>>>>>>>>>>> problem. In his test, w ith default security > >>>>>>>>>>>>> groups(which has remote security group), > >>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k > >>>>>>>>>>>>> Iptable rules > >>>>>> on evry > >>>>>>>>>>>>> compute node, although his patch can reduce > >>>>>>>>>>>>> the processing time, but > >>>>>> it don't > >>>>>>>>>>>>> solve this problem fundamentally. I have > >>>>>>>>>>>>> commit a BP to solve this > >>>>>> problem: > >>>>>>>>>>>>> > >>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security > >> > >>>>>> > >> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>> There are other people interested in this > >>>>>>>>>>>>> it? > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>> _______________________________________________ > >> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>> OpenStack-dev mailing list > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>> [email protected] >> > > > > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>> > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >>>>>> > >> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>> OpenStack-dev mailing list > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>> [email protected] >> > > >>>>>>>>> > >>>>>>>>>>> > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>>>> > >>>>>> > >>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>>> _______________________________________________ > >>>>>>>>> > >>>>>>>>>> OpenStack-dev mailing list > >>>>>>>>> > >>>>>>>>>> [email protected] >> > >>>>>>>>>> > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>>>> > >>>>>> > >>> > >>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> OpenStack-dev mailing list > >>>>>>>>> [email protected] >> > >>>>>>>>> > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>>>> > >>>>>> > >> > >>>>>>>> _______________________________________________ > >>>>>>>> OpenStack-dev mailing list > >>>>>>>> [email protected] > > >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >>>>>>>> > >>>>>>>> > >> > >>>>>>> > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> OpenStack-dev mailing list > >>>>>>> [email protected] > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >>>>>>> > >>>>>>> > >> > >>>>>>> > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> OpenStack-dev mailing list > >>>>>>> [email protected] > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >>>>>>> > >>>>>>> > >> > >> > >>>>>> _______________________________________________ > >>>>>> OpenStack-dev mailing list > >>>>>> [email protected] > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >>>>>> > >>>>> > >>>>> > >> > >> _______________________________________________ > >>>>> OpenStack-dev mailing list [email protected] > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>> > >> > >>>>> > >_______________________________________________ > >>>> OpenStack-dev mailing list [email protected] > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>> > >> > >>>> > >>>> > >>> _______________________________________________ OpenStack-dev > >>> mailing list [email protected] > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>>> > >>>>>_______________________________________________ > >>>>>OpenStack-dev > >>> > >mailing list > >>>>> [email protected] > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>> > >_______________________________________________ > >>>> OpenStack-dev mailing list [email protected] > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>> > >>> > >>>_______________________________________________ > >>>OpenStack-dev > >>>> > >mailing list > >>> [email protected] > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >>> > >> > >> > >> > >> > >> _______________________________________________ OpenStack-dev > >> mailing list [email protected] > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >-----BEGIN PGP SIGNATURE----- > >Version: GnuPG/MacGPG2 v2.0.22 (Darwin) > >Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > > >iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR > >IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE > >WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros > >7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23 > >1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di > >/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc= > >=TxAT > >-----END PGP SIGNATURE----- > > > >_______________________________________________ > >OpenStack-dev mailing list > >[email protected] > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
