On Wed, Jul 2, 2014 at 3:43 AM, Ihar Hrachyshka <[email protected]> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On 02/07/14 10:12, Miguel Angel Ajo wrote: >> >> Shihazhang, >> >> I really believe we need the RPC refactor done for this cycle, and >> given the close deadlines we have (July 10 for spec submission and >> July 20 for spec approval). >> >> Don't you think it's going to be better to split the work in >> several specs? >> >> 1) ipset optimization (you) 2) sg rpc optimization (without >> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you , >> me) >> >> >> This way we increase the chances of having part of this for the >> Juno cycle. If we go for something too complicated is going to take >> more time for approval. >> > > I agree. And it not only increases chances to get at least some of > those highly demanded performance enhancements to get into Juno, it's > also "the right thing to do" (c). It's counterproductive to put > multiple vaguely related enhancements in single spec. This would dim > review focus and put us into position of getting 'all-or-nothing'. We > can't afford that. > > Let's leave one spec per enhancement. @Shihazhang, what do you think? > +100
File these as separate specs, and lets see how much of this we can get into Juno. Thanks for taking this enhancement and performance improvement on everyone! Kyle >> >> Also, I proposed the details of "2", trying to bring awareness on >> the topic, as I have been working with the scale lab in Red Hat to >> find and understand those issues, I have a very good knowledge of >> the problem and I believe I could make a very fast advance on the >> issue at the RPC level. >> >> Given that, I'd like to work on this specific part, whether or not >> we split the specs, as it's something we believe critical for >> neutron scalability and thus, *nova parity*. >> >> I will start a separate spec for "2", later on, if you find it ok, >> we keep them as separate ones, if you believe having just 1 spec >> (for 1 & 2) is going be safer for juno-* approval, then we can >> incorporate my spec in yours, but then "add-ipset-to-security" is >> not a good spec title to put all this together. >> >> >> Best regards, Miguel Ángel. >> >> >> On 07/02/2014 03:37 AM, shihanzhang wrote: >>> >>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my spes, >>> but I will also optimization the RPC from security group agent to >>> neutron server. Now the modle is 'port[rule1,rule2...], port...', >>> I will change it to 'port[sg1, sg2..]', this can reduce the size >>> of RPC respose message from neutron server to security group >>> agent. >>> >>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo" >>> <[email protected]> wrote: >>>> >>>> >>>> Ok, I was talking with Édouard @ IRC, and as I have time to >>>> work into this problem, I could file an specific spec for the >>>> security group RPC optimization, a masterplan in two steps: >>>> >>>> 1) Refactor the current RPC communication for >>>> security_groups_for_devices, which could be used for full >>>> syncs, etc.. >>>> >>>> 2) Benchmark && make use of a fanout queue per security group >>>> to make sure only the hosts with instances on a certain >>>> security group get the updates as they happen. >>>> >>>> @shihanzhang do you find it reasonable? >>>> >>>> >>>> >>>> ----- Original Message ----- >>>>> ----- Original Message ----- >>>>>> @Nachi: Yes that could a good improvement to factorize the >>>>>> RPC >>>>> mechanism. >>>>>> >>>>>> Another idea: What about creating a RPC topic per security >>>>>> group (quid of the >>>>> RPC topic >>>>>> scalability) on which an agent subscribes if one of its >>>>>> ports is >>>>> associated >>>>>> to the security group? >>>>>> >>>>>> Regards, Édouard. >>>>>> >>>>>> >>>>> >>>>> >>>>> Hmm, Interesting, >>>>> >>>>> @Nachi, I'm not sure I fully understood: >>>>> >>>>> >>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] .. >>>>> port[SG_ID1, SG_ID2], port2 , port3 >>>>> >>>>> >>>>> Probably we may need to include also the SG_IP_LIST = >>>>> [SG_IP1, SG_IP2] ... >>>>> >>>>> >>>>> and let the agent do all the combination work. >>>>> >>>>> Something like this could make sense? >>>>> >>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....], >>>>> SG2:{IPs:[....],RULES:[....]} } >>>>> >>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... } >>>>> >>>>> >>>>> @Edouard, actually I like the idea of having the agent >>>>> subscribed to security groups they have ports on... That >>>>> would remove the need to include all the security groups >>>>> information on every call... >>>>> >>>>> But would need another call to get the full information of a >>>>> set of security groups at start/resync if we don't already >>>>> have any. >>>>> >>>>> >>>>>> >>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang < >>>>> [email protected] > >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> hi Miguel Ángel, I am very agree with you about the >>>>>> following point: >>>>>>> * physical implementation on the hosts (ipsets, nftables, >>>>>>> ... ) >>>>>> --this can reduce the load of compute node. >>>>>>> * rpc communication mechanisms. >>>>>> -- this can reduce the load of neutron server can you help >>>>>> me to review my BP specs? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" < >>>>> [email protected] > >>>>>> wrote: >>>>>>> >>>>>>> Hi it's a very interesting topic, I was getting ready to >>>>>>> raise the same concerns about our security groups >>>>>>> implementation, >>>>> shihanzhang >>>>>>> thank you for starting this topic. >>>>>>> >>>>>>> Not only at low level where (with our default security >>>>>>> group rules -allow all incoming from 'default' sg- the >>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the >>>>>>> "security_group_rules_for_devices" rpc call from >>>>>>> ovs-agent to neutron-server grows to message sizes of >>>>>>>> 100MB, >>>>>>> generating serious scalability issues or timeouts/retries >>>>>>> that totally break neutron service. >>>>>>> >>>>>>> (example trace of that RPC call with a few instances >>>>>>> http://www.fpaste.org/104401/14008522/ ) >>>>>>> >>>>>>> I believe that we also need to review the RPC calling >>>>>>> mechanism for the OVS agent here, there are several >>>>>>> possible approaches to >>>>> breaking >>>>>>> down (or/and CIDR compressing) the information we return >>>>>>> via this >>>>> api >>>>>>> call. >>>>>>> >>>>>>> >>>>>>> So we have to look at two things here: >>>>>>> >>>>>>> * physical implementation on the hosts (ipsets, nftables, >>>>>>> ... ) * rpc communication mechanisms. >>>>>>> >>>>>>> Best regards, Miguel Ángel. >>>>>>> >>>>>>> ----- Mensaje original ----- >>>>>>> >>>>>>>> Do you though about nftables that will replace >>>>> {ip,ip6,arp,eb}tables? >>>>>>>> It also based on the rule set mechanism. The issue in >>>>>>>> that proposition, it's only stable since the begin >>>>> of the >>>>>>>> year and on Linux kernel 3.13. But there lot of pros I >>>>>>>> don't list here (leverage iptables >>>>> limitation, >>>>>>>> efficient update rule, rule set, standardization of >>>>>>>> netfilter commands...). >>>>>>> >>>>>>>> Édouard. >>>>>>> >>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < >>>>>>>> [email protected] > wrote: >>>>>>> >>>>>>>>> we have done some tests, but have different result: >>>>>>>>> the >>>>> performance is >>>>>>>>> nearly the same for empty and 5k rules in iptable, >>>>>>>>> but huge gap between enable/disable iptable hook on >>>>>>>>> linux bridge >>>>>>>> >>>>>>> >>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < >>>>> [email protected] >>>>>>>>>> >>>>>>>>> wrote: >>>>>>>> >>>>>>> >>>>>>>>>> Now I have not get accurate test data, but I can >>>>>>>>>> confirm the following points： >>>>>>>>> >>>>>>>> >>>>>>>>>> 1. In compute node, the iptable's chain of a VM is >>>>>>>>>> liner, >>>>> iptable >>>>>>>>>> filter it one by one, if a VM in default security >>>>>>>>>> group and this default security group have many >>>>>>>>>> members, but ipset chain is set, the time >>>>> ipset >>>>>>>>>> filter one and many member is not much difference. >>>>>>>>> >>>>>>>> >>>>>>>>>> 2. when the iptable rule is very large, the >>>>>>>>>> probability of >>>>> failure >>>>>>>>>> that iptable-save save the iptable rule is very >>>>>>>>>> large. >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" < >>>>>>>>>> [email protected] >>>>>> wrote: >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>>>>>> This sounds like a good idea to handle some of >>>>>>>>>>> the >>>>> performance >>>>>>>>>>> issues until the ovs firewall can be implemented >>>>>>>>>>> down the the line. >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>>>> Do you have any performance comparisons? >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" < >>>>> [email protected] > >>>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>>>>>>> Hello all, >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>>>>>>> Now in neutron, it use iptable implementing >>>>>>>>>>>> security >>>>> group, but >>>>>>>>>>>> the performance of this implementation is very >>>>>>>>>>>> poor, there >>>>> is a bug: >>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272 >>>>>>>>>>>> to >>>>> reflect this >>>>>>>>>>>> problem. In his test, w ith default security >>>>>>>>>>>> groups(which has remote security group), beyond >>>>>>>>>>>> 250-300 VMs, there were around 6k Iptable >>>>>>>>>>>> rules >>>>> on evry >>>>>>>>>>>> compute node, although his patch can reduce the >>>>>>>>>>>> processing time, but >>>>> it don't >>>>>>>>>>>> solve this problem fundamentally. I have commit >>>>>>>>>>>> a BP to solve this >>>>> problem: >>>>>>>>>>>> >>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security >>>>> >>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>>>>> There are other people interested in this it? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>>>>> OpenStack-dev mailing list >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>>>>> [email protected] >> > > > >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>>>>> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>> >>>>>>>> >>>>>>>>>> OpenStack-dev mailing list >>>>>>>>> >>>>>>>> >>>>>>>>>> [email protected] >> > >>>>>>>> >>>>>>>>>> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >> >>>>>>>>>>> >>>>>>>> >>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>> >>>>>>>>> OpenStack-dev mailing list >>>>>>>> >>>>>>>>> [email protected] >> >>>>>>>>> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >> >>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> OpenStack-dev mailing list >>>>>>>> [email protected] >> >>>>>>>> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> > >>>>>>> _______________________________________________ >>>>>>> OpenStack-dev mailing list >>>>>>> [email protected] > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>>>> >> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> OpenStack-dev mailing list >>>>>> [email protected] >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>>> >> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> OpenStack-dev mailing list >>>>>> [email protected] >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>>> >> >>>>> >>>>> _______________________________________________ OpenStack-dev >>>>> mailing list [email protected] >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>> >>>> >>>>> > _______________________________________________ >>>> OpenStack-dev mailing list [email protected] >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> >>> >>> >>> >>>> > _______________________________________________ >>> OpenStack-dev mailing list [email protected] >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >>> >> _______________________________________________ OpenStack-dev >> mailing list [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.22 (Darwin) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQEcBAEBCgAGBQJTs8YsAAoJEC5aWaUY1u573G8H/jfRgoJMEJ0al9+io8bKtoLK > 1oznScn4StQGy+eObk4cTIY1qmfEBcqdZLsGXkVtDVxtJZo/lUwViAX/r0qffazB > dYCRoFqjhj+JhbQ2ul51A42vZ528lVdzwRdV0Hmwr9AdHdLwjdHi4msTqMXb0Tzs > HiLIp9IED758xDN9DfCv3BTFR1EqQoczrSKmFV7eUnS7A8dTun3LzhyptD9dwzR4 > jKl2RJ8/rpzu+oU0R/yju6IaF6Pe48D5UOH1LjNvL3BpUBj/ULH4Z/Sw8VSV9xPj > VtOowoB6evsK3bmehsiTXo/nn/M/B1Qo8n2Q+ox2A5q5/0Mhw8pH41GrOPECZlQ= > =/+cP > -----END PGP SIGNATURE----- > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
