On 08/20/2014 07:34 AM, Miguel Angel Ajo Pelayo wrote:
I couldn't resist making a little benchmark test of the new RPC implementation
shihanzhang wrote:

http://www.ajo.es/post/95269040924/neutron-security-group-rules-for-devices-rpc-rewrite

The results are awesome :-)

Indeed, fantastic news. ++

-jay

We yet need to polish the tests a bit, and it's ready.

Best regards,
Miguel Ángel.

----- Original Message -----
On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang <ayshihanzh...@126.com> wrote:

With the deployment 'nova + neutron + openvswitch', when we bulk create
about 500 VM with a default security group, the CPU usage of neutron-server
and openvswitch agent is very high, especially the CPU usage of openvswitch
agent will be 100%, this will cause creating VMs failed.

With the method discussed in mailist:

1) ipset optimization   (https://review.openstack.org/#/c/100761/)

3) sg rpc optimization (with fanout)
(https://review.openstack.org/#/c/104522/)

I have implement  these two scheme in my deployment,  when we again bulk
create about 500 VM with a default security group, the CPU usage of
openvswitch agent will reduce to 10%, even lower than 10%, so I think the
iprovement of these two options are very efficient.

Who can help us to review our spec?

This is great work! These are on my list of things to review in detail
soon, but given the Neutron sprint this week, I haven't had time yet.
I'll try to remedy that by the weekend.

Thanks!
Kyle

    Best regards,
shihanzhang





At 2014-07-03 10:08:21, "Ihar Hrachyshka" <ihrac...@redhat.com> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Oh, so you have the enhancement implemented? Great! Any numbers that
shows how much we gain from that?

/Ihar

On 03/07/14 02:49, shihanzhang wrote:
Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
I will modify my spec, when the spec is approved, I will commit the
codes as soon as possilbe!





At 2014-07-02 10:12:34, "Miguel Angel Ajo" <majop...@redhat.com>
wrote:

Nice Shihanzhang,

Do you mean the ipset implementation is ready, or just the
spec?.


For the SG group refactor, I don't worry about who does it, or
who takes the credit, but I believe it's important we address
this bottleneck during Juno trying to match nova's scalability.

Best regards, Miguel Ángel.


On 07/02/2014 02:50 PM, shihanzhang wrote:
hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
split  the work in several specs, I have finished the work (
ipset optimization), you can do 'sg rpc optimization (without
fanout)'. as the third part(sg rpc optimization (with fanout)),
I think we need talk about it, because just using ipset to
optimize security group agent codes does not bring the best
results!

Best regards, shihanzhang.








At 2014-07-02 04:43:24, "Ihar Hrachyshka" <ihrac...@redhat.com>
wrote:
On 02/07/14 10:12, Miguel Angel Ajo wrote:

Shihazhang,

I really believe we need the RPC refactor done for this cycle,
and given the close deadlines we have (July 10 for spec
submission and July 20 for spec approval).

Don't you think it's going to be better to split the work in
several specs?

1) ipset optimization   (you) 2) sg rpc optimization (without
fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
, me)


This way we increase the chances of having part of this for the
Juno cycle. If we go for something too complicated is going to
take more time for approval.


I agree. And it not only increases chances to get at least some of
those highly demanded performance enhancements to get into Juno,
it's also "the right thing to do" (c). It's counterproductive to
put multiple vaguely related enhancements in single spec. This
would dim review focus and put us into position of getting
'all-or-nothing'. We can't afford that.

Let's leave one spec per enhancement. @Shihazhang, what do you
think?


Also, I proposed the details of "2", trying to bring awareness
on the topic, as I have been working with the scale lab in Red
Hat to find and understand those issues, I have a very good
knowledge of the problem and I believe I could make a very fast
advance on the issue at the RPC level.

Given that, I'd like to work on this specific part, whether or
not we split the specs, as it's something we believe critical
for neutron scalability and thus, *nova parity*.

I will start a separate spec for "2", later on, if you find it
ok, we keep them as separate ones, if you believe having just 1
spec (for 1 & 2) is going be safer for juno-* approval, then we
can incorporate my spec in yours, but then
"add-ipset-to-security" is not a good spec title to put all this
together.


Best regards, Miguel Ángel.


On 07/02/2014 03:37 AM, shihanzhang wrote:

hi Miguel Angel Ajo Pelayo! I agree with you and modify my
spes, but I will also optimization the RPC from security group
agent to neutron server. Now the modle is
'port[rule1,rule2...], port...', I will change it to 'port[sg1,
sg2..]', this can reduce the size of RPC respose message from
neutron server to security group agent.

At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
<mangel...@redhat.com> wrote:


Ok, I was talking with Édouard @ IRC, and as I have time to
work into this problem, I could file an specific spec for
the security group RPC optimization, a masterplan in two
steps:

1) Refactor the current RPC communication for
security_groups_for_devices, which could be used for full
syncs, etc..

2) Benchmark && make use of a fanout queue per security
group to make sure only the hosts with instances on a
certain security group get the updates as they happen.

@shihanzhang do you find it reasonable?



----- Original Message -----
----- Original Message -----
@Nachi: Yes that could a good improvement to factorize
the RPC
mechanism.

Another idea: What about creating a RPC topic per
security group (quid of the
RPC topic
scalability) on which an agent subscribes if one of its
ports is
associated
to the security group?

Regards, Édouard.




Hmm, Interesting,

@Nachi, I'm not sure I fully understood:


SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
port[SG_ID1, SG_ID2], port2 , port3


Probably we may need to include also the SG_IP_LIST =
[SG_IP1, SG_IP2] ...


and let the agent do all the combination work.

Something like this could make sense?

Security_Groups = {SG1:{IPs:[....],RULES:[....],
SG2:{IPs:[....],RULES:[....]} }

Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }


@Edouard, actually I like the idea of having the agent
subscribed to security groups they have ports on... That
would remove the need to include all the security groups
information on every call...

But would need another call to get the full information of
a set of security groups at start/resync if we don't
already have any.



On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
ayshihanzh...@126.com >
wrote:



hi Miguel Ángel, I am very agree with you about the
following point:
* physical implementation on the hosts (ipsets,
nftables, ... )
--this can reduce the load of compute node.
* rpc communication mechanisms.
-- this can reduce the load of neutron server can you
help me to review my BP specs?







At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
mangel...@redhat.com >
wrote:

Hi it's a very interesting topic, I was getting ready
to raise the same concerns about our security groups
implementation,
shihanzhang
thank you for starting this topic.

Not only at low level where (with our default security
group rules -allow all incoming from 'default' sg- the
iptable rules will grow in ~X^2 for a tenant, and, the
"security_group_rules_for_devices" rpc call from
ovs-agent to neutron-server grows to message sizes of
100MB,
generating serious scalability issues or
timeouts/retries that totally break neutron service.

(example trace of that RPC call with a few instances
http://www.fpaste.org/104401/14008522/ )

I believe that we also need to review the RPC calling
mechanism for the OVS agent here, there are several
possible approaches to
breaking
down (or/and CIDR compressing) the information we
return via this
api
call.


So we have to look at two things here:

* physical implementation on the hosts (ipsets,
nftables, ... ) * rpc communication mechanisms.

Best regards, Miguel Ángel.

----- Mensaje original -----

Do you though about nftables that will replace
{ip,ip6,arp,eb}tables?
It also based on the rule set mechanism. The issue
in that proposition, it's only stable since the
begin
of the
year and on Linux kernel 3.13. But there lot of pros
I don't list here (leverage iptables
limitation,
efficient update rule, rule set, standardization of
netfilter commands...).

Édouard.

On Thu, Jun 19, 2014 at 8:25 AM, henry hly <
henry4...@gmail.com > wrote:

we have done some tests, but have different
result: the
performance is
nearly the same for empty and 5k rules in iptable,
but huge gap between enable/disable iptable hook
on linux bridge


On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
ayshihanzh...@126.com

wrote:


Now I have not get accurate test data, but I can
confirm the following points?


1. In compute node, the iptable's chain of a VM
is liner,
iptable
filter it one by one, if a VM in default
security group and this default security group
have many members, but ipset chain is set, the
time
ipset
filter one and many member is not much
difference.


2. when the iptable rule is very large, the
probability of
failure
that iptable-save save the iptable rule is very
large.



At 2014-06-19 10:55:56, "Kevin Benton" <
blak...@gmail.com
wrote:



This sounds like a good idea to handle some of
the
performance
issues until the ovs firewall can be
implemented down the the line.



Do you have any performance comparisons?



On Jun 18, 2014 7:46 PM, "shihanzhang" <
ayshihanzh...@126.com >
wrote:




Hello all,





Now in neutron, it use iptable implementing
security
group, but
the performance of this implementation is
very poor, there
is a bug:
https://bugs.launchpad.net/neutron/+bug/1302272


to
reflect this
problem. In his test, w ith default security
groups(which has remote security group),
beyond 250-300 VMs, there were around 6k
Iptable rules
on evry
compute node, although his patch can reduce
the processing time, but
it don't
solve this problem fundamentally. I have
commit a BP to solve this
problem:

https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security







There are other people interested in this
it?





_______________________________________________







OpenStack-dev mailing list




OpenStack-dev@lists.openstack.org >> > > >




http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev








_______________________________________________


OpenStack-dev mailing list


OpenStack-dev@lists.openstack.org >> >


http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev






_______________________________________________

OpenStack-dev mailing list

OpenStack-dev@lists.openstack.org >>

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org >>

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org >
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev






_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev






_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





_______________________________________________
OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev









_______________________________________________
OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




_______________________________________________ OpenStack-dev
mailing list OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_______________________________________________
OpenStack-dev

mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev







_______________________________________________
OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev

mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev






_______________________________________________ OpenStack-dev
mailing list OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR
IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE
WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros
7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23
1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di
/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc=
=TxAT
-----END PGP SIGNATURE-----

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to