Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-28 Thread Daniel Alvarez Sanchez
Thanks a lot Han! Great and swift work! I'm testing them now, will let you know ASAP. On Thu, Mar 1, 2018 at 8:39 AM, Han Zhou wrote: > > > On Mon, Feb 26, 2018 at 12:05 PM, Ben Pfaff wrote: > > > > On Fri, Feb 23, 2018 at 03:51:28PM -0800, Han Zhou wrote: > > > On Fri, Feb 23, 2018 at 2:17 PM,

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-28 Thread Han Zhou
On Mon, Feb 26, 2018 at 12:05 PM, Ben Pfaff wrote: > > On Fri, Feb 23, 2018 at 03:51:28PM -0800, Han Zhou wrote: > > On Fri, Feb 23, 2018 at 2:17 PM, Ben Pfaff wrote: > > > > > > On Tue, Feb 20, 2018 at 08:56:42AM -0800, Han Zhou wrote: > > > > On Tue, Feb 20, 2018 at 8:15 AM, Ben Pfaff wrote: >

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-26 Thread Ben Pfaff
On Fri, Feb 23, 2018 at 03:51:28PM -0800, Han Zhou wrote: > On Fri, Feb 23, 2018 at 2:17 PM, Ben Pfaff wrote: > > > > On Tue, Feb 20, 2018 at 08:56:42AM -0800, Han Zhou wrote: > > > On Tue, Feb 20, 2018 at 8:15 AM, Ben Pfaff wrote: > > > > > > > > On Mon, Feb 19, 2018 at 11:33:11AM +0100, Daniel

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-23 Thread Han Zhou
On Fri, Feb 23, 2018 at 2:17 PM, Ben Pfaff wrote: > > On Tue, Feb 20, 2018 at 08:56:42AM -0800, Han Zhou wrote: > > On Tue, Feb 20, 2018 at 8:15 AM, Ben Pfaff wrote: > > > > > > On Mon, Feb 19, 2018 at 11:33:11AM +0100, Daniel Alvarez Sanchez wrote: > > > > @Han, I can try rebase the patch if you

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-23 Thread Ben Pfaff
On Tue, Feb 20, 2018 at 08:56:42AM -0800, Han Zhou wrote: > On Tue, Feb 20, 2018 at 8:15 AM, Ben Pfaff wrote: > > > > On Mon, Feb 19, 2018 at 11:33:11AM +0100, Daniel Alvarez Sanchez wrote: > > > @Han, I can try rebase the patch if you want but that was > > > basically renaming the Address_Set tab

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-20 Thread Daniel Alvarez
> On 20 Feb 2018, at 19:50, Lucas Alvares Gomes wrote: > > Hi, > >> On Tue, Feb 20, 2018 at 4:56 PM, Han Zhou wrote: >> >> >>> On Tue, Feb 20, 2018 at 8:15 AM, Ben Pfaff wrote: >>> On Mon, Feb 19, 2018 at 11:33:11AM +0100, Daniel Alvarez Sanchez wrote: @Han, I can try rebase th

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-20 Thread Lucas Alvares Gomes
Hi, On Tue, Feb 20, 2018 at 4:56 PM, Han Zhou wrote: > > > On Tue, Feb 20, 2018 at 8:15 AM, Ben Pfaff wrote: >> >> On Mon, Feb 19, 2018 at 11:33:11AM +0100, Daniel Alvarez Sanchez wrote: >> > @Han, I can try rebase the patch if you want but that was >> > basically renaming the Address_Set table

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-20 Thread Han Zhou
On Tue, Feb 20, 2018 at 8:15 AM, Ben Pfaff wrote: > > On Mon, Feb 19, 2018 at 11:33:11AM +0100, Daniel Alvarez Sanchez wrote: > > @Han, I can try rebase the patch if you want but that was > > basically renaming the Address_Set table and from Ben's > > comment, it may be better to keep the name. No

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-20 Thread Ben Pfaff
On Mon, Feb 19, 2018 at 11:33:11AM +0100, Daniel Alvarez Sanchez wrote: > @Han, I can try rebase the patch if you want but that was > basically renaming the Address_Set table and from Ben's > comment, it may be better to keep the name. Not sure, > however, how we can proceed to address Lucas' point

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-19 Thread Daniel Alvarez Sanchez
On Tue, Feb 13, 2018 at 8:32 PM, Ben Pfaff wrote: > On Tue, Feb 13, 2018 at 12:39:56PM +0100, Daniel Alvarez Sanchez wrote: > > Hi folks, > > > > As we're doing some performance tests in OpenStack using OVN, > > we noticed that as we keep creating ports, the time for creating a > > single port in

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-19 Thread Daniel Alvarez Sanchez
Just for completeness, I did some tests with and without the JSON parser C extension [0]. There's no significant gain (at this point), maybe when we add the port groups it's more noticeable though. Average time without it's been 2.09s while with the C extension it's been 2.03s. @Han, I can try reb

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-16 Thread Han Zhou
Hi Daniel, Thanks for the detailed profiling! On Fri, Feb 16, 2018 at 6:50 AM, Daniel Alvarez Sanchez wrote: > > About the duplicated processing of the update2 messages, I've verified that those are not always present. I've isolated the scenario further and did tcpdump and debugging on the exact

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-16 Thread Daniel Alvarez Sanchez
About the duplicated processing of the update2 messages, I've verified that those are not always present. I've isolated the scenario further and did tcpdump and debugging on the exact process which is sending the'transact' command and I see no update2 processing duplicates. Among the rest of the wo

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-16 Thread Daniel Alvarez Sanchez
On Fri, Feb 16, 2018 at 12:12 PM, Daniel Alvarez Sanchez < dalva...@redhat.com> wrote: > I've found out more about what is running slow in this scenario. > I've profiled the processing of the update2 messages and here you can > see the sequence of calls to __process_update2 (idl.py) when I'm > cre

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-16 Thread Daniel Alvarez Sanchez
I've found out more about what is running slow in this scenario. I've profiled the processing of the update2 messages and here you can see the sequence of calls to __process_update2 (idl.py) when I'm creating a new port via OpenStack on a system loaded with 800 ports on the same Logical Switch: 1

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-15 Thread Ben Pfaff
On Thu, Feb 15, 2018 at 01:38:04PM -0800, Han Zhou wrote: > On Thu, Feb 15, 2018 at 12:56 PM, Ben Pfaff wrote: > > > > On Thu, Feb 15, 2018 at 06:50:15PM +, Lucas Alvares Gomes wrote: > > > Hi all, > > > > > > We currently have a problem with Address_Set in networking-ovn (and > > > potentiall

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-15 Thread Daniel Alvarez Sanchez
On Wed, Feb 14, 2018 at 9:34 PM, Han Zhou wrote: > > > On Wed, Feb 14, 2018 at 9:45 AM, Ben Pfaff wrote: > > > > On Wed, Feb 14, 2018 at 11:27:11AM +0100, Daniel Alvarez Sanchez wrote: > > > Thanks for your inputs. I need to look more carefully into the patch > you > > > submitted but it looks l

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-15 Thread Han Zhou
On Thu, Feb 15, 2018 at 12:56 PM, Ben Pfaff wrote: > > On Thu, Feb 15, 2018 at 06:50:15PM +, Lucas Alvares Gomes wrote: > > Hi all, > > > > We currently have a problem with Address_Set in networking-ovn (and > > potentially other technologies using OVN as backend) which *maybe* > > could be fi

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-15 Thread Ben Pfaff
On Thu, Feb 15, 2018 at 06:50:15PM +, Lucas Alvares Gomes wrote: > Hi all, > > We currently have a problem with Address_Set in networking-ovn (and > potentially other technologies using OVN as backend) which *maybe* > could be fixed together with this idea of a new "port set" (a.k.a > macro se

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-15 Thread Lucas Alvares Gomes
Hi all, We currently have a problem with Address_Set in networking-ovn (and potentially other technologies using OVN as backend) which *maybe* could be fixed together with this idea of a new "port set" (a.k.a macro set). The problem is bit tricky but it shows as a race condition between creating

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Ben Pfaff
On Thu, Feb 15, 2018 at 12:36:35AM +0100, Daniel Alvarez Sanchez wrote: > If we would have the Port_Set we could simply write the match part as > "outport == $security_group1 && ip4 && ip4.src == 0.0.0.0/0 && tcp && > tcp.dst == 22" > and reduce the number of ACLs to 1 per security group rule inste

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Ben Pfaff
On Wed, Feb 14, 2018 at 03:45:21PM -0800, Han Zhou wrote: > On Wed, Feb 14, 2018 at 3:39 PM, Ben Pfaff wrote: > > > > On Wed, Feb 14, 2018 at 03:29:34PM -0800, Han Zhou wrote: > > > On Wed, Feb 14, 2018 at 3:08 PM, Ben Pfaff wrote: > > > > > > > > On Wed, Feb 14, 2018 at 02:25:56PM -0800, Han Zho

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Han Zhou
On Wed, Feb 14, 2018 at 3:39 PM, Ben Pfaff wrote: > > On Wed, Feb 14, 2018 at 03:29:34PM -0800, Han Zhou wrote: > > On Wed, Feb 14, 2018 at 3:08 PM, Ben Pfaff wrote: > > > > > > On Wed, Feb 14, 2018 at 02:25:56PM -0800, Han Zhou wrote: > > > > On Wed, Feb 14, 2018 at 1:40 PM, Ben Pfaff wrote: >

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Ben Pfaff
On Wed, Feb 14, 2018 at 03:29:34PM -0800, Han Zhou wrote: > On Wed, Feb 14, 2018 at 3:08 PM, Ben Pfaff wrote: > > > > On Wed, Feb 14, 2018 at 02:25:56PM -0800, Han Zhou wrote: > > > On Wed, Feb 14, 2018 at 1:40 PM, Ben Pfaff wrote: > > > > > > > > On Wed, Feb 14, 2018 at 12:34:19PM -0800, Han Zho

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Daniel Alvarez Sanchez
Ok let me paste some example but feel free to ask for any further details. 1 Logical Switch with 5 ports and 8 ACLs per port: # ovn-nbctl show switch c1fac5d4-b682-4078-9282-61cfa6383893 (neutron-d35e99a5-d9e9-4bc5-9ad4-08e0941f1820) (aka test_net) port 8a8be79b-7a24-4a19-b952-c68d839e0164 (

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Han Zhou
On Wed, Feb 14, 2018 at 3:08 PM, Ben Pfaff wrote: > > On Wed, Feb 14, 2018 at 02:25:56PM -0800, Han Zhou wrote: > > On Wed, Feb 14, 2018 at 1:40 PM, Ben Pfaff wrote: > > > > > > On Wed, Feb 14, 2018 at 12:34:19PM -0800, Han Zhou wrote: > > > > I remember there was a patch for ACL group in OVN, so

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Ben Pfaff
On Wed, Feb 14, 2018 at 02:25:56PM -0800, Han Zhou wrote: > On Wed, Feb 14, 2018 at 1:40 PM, Ben Pfaff wrote: > > > > On Wed, Feb 14, 2018 at 12:34:19PM -0800, Han Zhou wrote: > > > I remember there was a patch for ACL group in OVN, so that instead of > R*P > > > rows we will have only R + P rows,

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Daniel Alvarez Sanchez
Thanks a lot Han and Ben for looking into this! On Wed, Feb 14, 2018 at 9:34 PM, Han Zhou wrote: > > > On Wed, Feb 14, 2018 at 9:45 AM, Ben Pfaff wrote: > > > > On Wed, Feb 14, 2018 at 11:27:11AM +0100, Daniel Alvarez Sanchez wrote: > > > Thanks for your inputs. I need to look more carefully in

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Han Zhou
On Wed, Feb 14, 2018 at 1:40 PM, Ben Pfaff wrote: > > On Wed, Feb 14, 2018 at 12:34:19PM -0800, Han Zhou wrote: > > I remember there was a patch for ACL group in OVN, so that instead of R*P > > rows we will have only R + P rows, but didn't see it went through. > > I don't remember that. Any chanc

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Ben Pfaff
On Wed, Feb 14, 2018 at 12:34:19PM -0800, Han Zhou wrote: > I remember there was a patch for ACL group in OVN, so that instead of R*P > rows we will have only R + P rows, but didn't see it went through. I don't remember that. Any chance you could point me to it? > Is this also a good use case of

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Han Zhou
On Wed, Feb 14, 2018 at 9:45 AM, Ben Pfaff wrote: > > On Wed, Feb 14, 2018 at 11:27:11AM +0100, Daniel Alvarez Sanchez wrote: > > Thanks for your inputs. I need to look more carefully into the patch you > > submitted but it looks like, at least, we'll be reducing the number of > > calls to Datum._

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Ben Pfaff
On Wed, Feb 14, 2018 at 11:27:11AM +0100, Daniel Alvarez Sanchez wrote: > Thanks for your inputs. I need to look more carefully into the patch you > submitted but it looks like, at least, we'll be reducing the number of > calls to Datum.__cmp__ which should be good. Thanks. Please do take a look.

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-14 Thread Daniel Alvarez Sanchez
Hi Ben, Thanks for your inputs. I need to look more carefully into the patch you submitted but it looks like, at least, we'll be reducing the number of calls to Datum.__cmp__ which should be good. For the rest of the things, let me answer inline. On Tue, Feb 13, 2018 at 8:32 PM, Ben Pfaff wrot

Re: [ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-13 Thread Ben Pfaff
On Tue, Feb 13, 2018 at 12:39:56PM +0100, Daniel Alvarez Sanchez wrote: > Hi folks, > > As we're doing some performance tests in OpenStack using OVN, > we noticed that as we keep creating ports, the time for creating a > single port increases. Also, ovn-northd CPU consumption is quite > high (see

[ovs-discuss] OpenStack profiling with networking-ovn - port creation is slow

2018-02-13 Thread Daniel Alvarez Sanchez
Hi folks, As we're doing some performance tests in OpenStack using OVN, we noticed that as we keep creating ports, the time for creating a single port increases. Also, ovn-northd CPU consumption is quite high (see [0] which shows the CPU consumption when creating 1000 ports and deleting them. Last