Thanks a lot Han and Ben for looking into this! On Wed, Feb 14, 2018 at 9:34 PM, Han Zhou <[email protected]> wrote:
> > > On Wed, Feb 14, 2018 at 9:45 AM, Ben Pfaff <[email protected]> wrote: > > > > On Wed, Feb 14, 2018 at 11:27:11AM +0100, Daniel Alvarez Sanchez wrote: > > > Thanks for your inputs. I need to look more carefully into the patch > you > > > submitted but it looks like, at least, we'll be reducing the number of > > > calls to Datum.__cmp__ which should be good. > > > > Thanks. Please do take a look. It's a micro-optimization but maybe > > it'll help? > > > > > I probably didn't explain it very well. Right now we have N processes > > > for Neutron server (in every node). Each of those opens a connection > > > to NB db and they subscribe to updates from certain tables. Each time > > > a change happens, ovsdb-server will send N update2 messages that has > > > to be processed in this "expensive" way by each of those N > > > processes. My proposal (yet to be refined) would be to now open N+1 > > > connections to ovsdb-server and only subscribe to notifications from 1 > > > of those. So every time a new change happens, ovsdb-server will send 1 > > > update2 message. This message will be processed (using Py IDL as we do > > > now) and once processed, send it (mcast maybe?) to the rest N > > > processes. This msg could be simply a Python object serialized and > > > we'd be saving all this Datum, Atom, etc. processing by doing it just > > > once. > > > Daniel, I understand that the update2 messages sending would consume NB > ovsdb-server CPU and processing those update would consume neutron server > process CPU. However, are we sure it is the bottleneck for port creation? > No, I don't think this is the bottleneck at this stage with only 800 ports being created at that pace. As you say, I'm also not keen on doing such a big change right now but it's something to bear in mind for the future as ~50% of total processing time being spent in data.py/uuid.py is not neglectable. At least, it'd be nice to try Ben's suggestions and I'll do it as soon as I can. > > From ovsdb-server point of view, sending updates to tens of clients should > not be the bottleneck, considering that we have a lot more clients on HVs > for SB ovsdb-server. > Same would go for SB ovsdb-server though but yeah... I agree. > > From clients point of view, I think it is more of memory overhead than > CPU, and it also depends on how many neutron processes are running on the > same node. I didn't find neutron process CPU in your charts. I am hesitate > for such big change before we are clear about the bottleneck. The chart of > port creation time is very nice, but do we know which part of code > contributed to the linear growth? Do we have profiling for the time spent > in ovn_client.add_acls()? > This is my next step. It seems pretty clear that ACLs are the bottleneck. As a note, unsurprisingly, same goes for port deletion: $ date; for i in {1..800}; do openstack port delete port$i; done; date Wed Feb 14 17:45:07 UTC 2018 Wed Feb 14 19:36:09 UTC 2018 Same behavior shown during deletion as can be seen at [0]. [0] https://imgur.com/a/Dam10 > > OK. It's an optimization that does the work in one place rather than N > > places, so definitely a win from a CPU cost point of view, but it trades > > performance for increased complexity. It sounds like performance is > > really important so maybe the increased complexity is a fair trade. > > > > We might also be able to improve performance by using native code for > > some of the work. Were these tests done with the native code JSON > > parser that comes with OVS? It is dramatically faster than the Python > > code. > Yes, these tests were done using the native JSON parser. I saw that Python parsing had a 5% over the total execution time so recompiling with the --enabled-shared made it use the native parser. However it didn't make a big difference in the port creation times. In my first tests I also saw some reconnections from clients to ovsdb-server especially when the number of ports increased. (not many though, just a couple of them) during the 800 ports. I then changed the inactivity probe to 180s for both NB and SB connections but still the results were pretty much the same even though I didn't see any reconnections this time. > > > > On Tue, Feb 13, 2018 at 8:32 PM, Ben Pfaff <[email protected]> wrote: > > > > > > > Can you sketch the rows that are being inserted or modified when a > port > > > > is added? I would expect something like this as a minimum: > > > > > > > > * Insert one Logical_Switch_Port row. > > > > > > > > * Add pointer to Logical_Switch_Port to ports column in one > row > > > > in Logical_Switch. > > > > > > > > In addition it sounds like currently we're seeing: > > > > > > > > * Add one ACL row per security group rule. > > > > > > > > * Add pointers to ACL rows to acls column in one row in > > > > Logical_Switch. > > > > > > > This is what happens when we create a port in OpenStack (without > > > binding it) which belongs to a SG which allows ICMP and SSH traffic > > > and drops the rest [0] > > > > > > Basically, you were right and only thing missing was adding the new > > > address to the Address_Set table. > > > > OK. > > > > It sounds like the real scaling problem here is that for R security > > group rules and P ports, we have R*P rows in the ACL table. Is that > > correct? Should we aim to solve that problem? > > I think this might be the most valuable point to optimize for the > create_port scenario from Neutron. > I remember there was a patch for ACL group in OVN, so that instead of R*P > rows we will have only R + P rows, but didn't see it went through. > Is this also a good use case of conjuncture? > > > _______________________________________________ > > discuss mailing list > > [email protected] > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > >
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
