Re: [openstack-dev] [neutron]Performance of security group

2014-08-21 Thread Kyle Mestery
This is great work! Looking forward to seeing this get reviewed and merged in Juno! Kyle On Thu, Aug 21, 2014 at 6:49 AM, Édouard Thuleau wrote: > Nice job! That's awesome. > > Thanks, > Édouard. > > > On Thu, Aug 21, 2014 at 8:02 AM, Miguel Angel Ajo Pelayo > wrote: >> >> Thank you shihanzhang

Re: [openstack-dev] [neutron]Performance of security group

2014-08-21 Thread Édouard Thuleau
Nice job! That's awesome. Thanks, Édouard. On Thu, Aug 21, 2014 at 8:02 AM, Miguel Angel Ajo Pelayo < mangel...@redhat.com> wrote: > Thank you shihanzhang!, > > I can't believe I didn't realize the ipset part spec was accepted I live > on my own bubble... I will be reviewing and testing/helping

Re: [openstack-dev] [neutron]Performance of security group

2014-08-20 Thread Miguel Angel Ajo Pelayo
Thank you shihanzhang!, I can't believe I didn't realize the ipset part spec was accepted I live on my own bubble... I will be reviewing and testing/helping on that part too during the next few days, I was too concentrated in the RPC part. Best regards, - Original Message - > hi neutr

Re: [openstack-dev] [neutron]Performance of security group

2014-08-20 Thread shihanzhang
hi neutroner! my patch about BP:https://blueprints.launchpad.net/openstack/?searchtext=add-ipset-to-security need install ipset in devstack, I have commit the patch:https://review.openstack.org/#/c/113453/, who can help me review it, thanks very much! Best regards, shihanzhang A

Re: [openstack-dev] [neutron]Performance of security group

2014-08-20 Thread Martinx - ジェームズ
+1 "NFTablesDriver"! Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example: https://home.regit.org/2014/02/suricata-and-nftables/ Then, I'm wondering here... What benefits might come for OpenStack Nova / Neutron, if it comes with a NFTables driver, instead of the current IPTable

Re: [openstack-dev] [neutron]Performance of security group

2014-08-20 Thread Baohua Yang
Great! We met similar problems. The current mechanisms produce too many iptables rules, and it's hard to debug. Really look forward to seeing a more efficient security group design. On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery wrote: > On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang > wrote: >

Re: [openstack-dev] [neutron]Performance of security group

2014-08-20 Thread Jay Pipes
On 08/20/2014 07:34 AM, Miguel Angel Ajo Pelayo wrote: I couldn't resist making a little benchmark test of the new RPC implementation shihanzhang wrote: http://www.ajo.es/post/95269040924/neutron-security-group-rules-for-devices-rpc-rewrite The results are awesome :-) Indeed, fantastic news.

Re: [openstack-dev] [neutron]Performance of security group

2014-08-20 Thread Miguel Angel Ajo Pelayo
I couldn't resist making a little benchmark test of the new RPC implementation shihanzhang wrote: http://www.ajo.es/post/95269040924/neutron-security-group-rules-for-devices-rpc-rewrite The results are awesome :-) We yet need to polish the tests a bit, and it's ready. Best regards, Miguel Ángel

Re: [openstack-dev] [neutron]Performance of security group

2014-07-10 Thread Kyle Mestery
On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang wrote: > > With the deployment 'nova + neutron + openvswitch', when we bulk create > about 500 VM with a default security group, the CPU usage of neutron-server > and openvswitch agent is very high, especially the CPU usage of openvswitch > agent will b

Re: [openstack-dev] [neutron]Performance of security group

2014-07-10 Thread Miguel Angel Ajo Pelayo
Wow Shihanzhang, truly awesome!, Is the current implementation for https://review.openstack.org/#/c/104522/ also ready?, could you make it available?. Good work!, - Original Message - > > With the deployment 'nova + neutron + openvswitch', when we bulk create about > 500 VM with a

Re: [openstack-dev] [neutron]Performance of security group

2014-07-03 Thread Ihar Hrachyshka
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Oh, so you have the enhancement implemented? Great! Any numbers that shows how much we gain from that? /Ihar On 03/07/14 02:49, shihanzhang wrote: > Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today > I will modify my spec, when t

Re: [openstack-dev] [neutron]Performance of security group

2014-07-03 Thread Miguel Angel Ajo
I have created a separate spec for the RPC part. https://review.openstack.org/104522 On 07/02/2014 05:52 PM, Kyle Mestery wrote: On Wed, Jul 2, 2014 at 3:43 AM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 02/07/14 10:12, Miguel Angel Ajo wrote: Shihazhang, I

Re: [openstack-dev] [neutron]Performance of security group

2014-07-02 Thread Kyle Mestery
On Wed, Jul 2, 2014 at 3:43 AM, Ihar Hrachyshka wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > On 02/07/14 10:12, Miguel Angel Ajo wrote: >> >> Shihazhang, >> >> I really believe we need the RPC refactor done for this cycle, and >> given the close deadlines we have (July 10 for spe

Re: [openstack-dev] [neutron]Performance of security group

2014-07-02 Thread Miguel Angel Ajo
Nice Shihanzhang, Do you mean the ipset implementation is ready, or just the spec?. For the SG group refactor, I don't worry about who does it, or who takes the credit, but I believe it's important we address this bottleneck during Juno trying to match nova's scalability. Best regards,

Re: [openstack-dev] [neutron]Performance of security group

2014-07-02 Thread shihanzhang
hi Miguel Ángel and Ihar Hrachyshka, I agree with you that split the work in several specs, I have finished the work ( ipset optimization), you can do 'sg rpc optimization (without fanout)'. as the third part(sg rpc optimization (with fanout)), I think we need talk about it, because just using

Re: [openstack-dev] [neutron]Performance of security group

2014-07-02 Thread Ihar Hrachyshka
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 02/07/14 10:12, Miguel Angel Ajo wrote: > > Shihazhang, > > I really believe we need the RPC refactor done for this cycle, and > given the close deadlines we have (July 10 for spec submission and > July 20 for spec approval). > > Don't you thi

Re: [openstack-dev] [neutron]Performance of security group

2014-07-02 Thread Miguel Angel Ajo
Shihazhang, I really believe we need the RPC refactor done for this cycle, and given the close deadlines we have (July 10 for spec submission and July 20 for spec approval). Don't you think it's going to be better to split the work in several specs? 1) ipset optimization (you) 2) s

Re: [openstack-dev] [neutron]Performance of security group

2014-07-01 Thread shihanzhang
hi Miguel Angel Ajo Pelayo! I agree with you and modify my spes, but I will also optimization the RPC from security group agent to neutron server. Now the modle is 'port[rule1,rule2...], port...', I will change it to 'port[sg1, sg2..]', this can reduce the size of RPC respose message from neu

Re: [openstack-dev] [neutron]Performance of security group

2014-07-01 Thread Miguel Angel Ajo Pelayo
Ok, I was talking with Édouard @ IRC, and as I have time to work into this problem, I could file an specific spec for the security group RPC optimization, a masterplan in two steps: 1) Refactor the current RPC communication for security_groups_for_devices, which could be used for full syncs,

Re: [openstack-dev] [neutron]Performance of security group

2014-06-26 Thread Miguel Angel Ajo Pelayo
- Original Message - > @Nachi: Yes that could a good improvement to factorize the RPC mechanism. > > Another idea: > What about creating a RPC topic per security group (quid of the RPC topic > scalability) on which an agent subscribes if one of its ports is associated > to the security gro

Re: [openstack-dev] [neutron]Performance of security group

2014-06-23 Thread Édouard Thuleau
@Nachi: Yes that could a good improvement to factorize the RPC mechanism. Another idea: What about creating a RPC topic per security group (quid of the RPC topic scalability) on which an agent subscribes if one of its ports is associated to the security group? Regards, Édouard. On Fri, Jun 20,

Re: [openstack-dev] [neutron]Performance of security group

2014-06-19 Thread shihanzhang
hi Miguel Ángel, I am very agree with you about the following point: > * physical implementation on the hosts (ipsets, nftables, ... ) --this can reduce the load of compute node. > * rpc communication mechanisms. --this can reduce the load of neutron server can you help me to review my BP s

Re: [openstack-dev] [neutron]Performance of security group

2014-06-19 Thread Nachi Ueno
Hi folks Thank you for your starting this topic. Let me share some of my ideas (1) Improve security_group_rules_for_devices In current implementation, we are generating rules per port in server side. It is something like this. port1[SG_Rule1, SG_Rule2, SG_Rule3] .. port2, port3 This can be imp

Re: [openstack-dev] [neutron]Performance of security group

2014-06-19 Thread Miguel Angel Ajo Pelayo
Hi it's a very interesting topic, I was getting ready to raise the same concerns about our security groups implementation, shihanzhang thank you for starting this topic. Not only at low level where (with our default security group rules -allow all incoming from 'default' sg- the iptable rules

Re: [openstack-dev] [neutron]Performance of security group

2014-06-19 Thread Édouard Thuleau
Do you though about nftables that will replace {ip,ip6,arp,eb}tables? It also based on the rule set mechanism. The issue in that proposition, it's only stable since the begin of the year and on Linux kernel 3.13. But there lot of pros I don't list here (leverage iptables limitation, efficient updat

Re: [openstack-dev] [neutron]Performance of security group

2014-06-18 Thread henry hly
we have done some tests, but have different result: the performance is nearly the same for empty and 5k rules in iptable, but huge gap between enable/disable iptable hook on linux bridge On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang wrote: > Now I have not get accurate test data, but I can con

Re: [openstack-dev] [neutron]Performance of security group

2014-06-18 Thread shihanzhang
Now I have not get accurate test data, but I can confirm the following points: 1. In compute node, the iptable's chain of a VM is liner, iptable filter it one by one, if a VM in default security group and this default security group have many members, but ipset chain is set, the time ipset filte

Re: [openstack-dev] [neutron]Performance of security group

2014-06-18 Thread Kevin Benton
This sounds like a good idea to handle some of the performance issues until the ovs firewall can be implemented down the the line. Do you have any performance comparisons? On Jun 18, 2014 7:46 PM, "shihanzhang" wrote: > Hello all, > > Now in neutron, it use iptable implementing security group, bu