Re: [ovs-dev] [patch_v8] ovn: Add datapaths of interest filtering.

[Darrell Ball]

On 12/1/16, 8:50 PM, "ovs-dev-boun...@openvswitch.org on behalf of Mickey 
Spiegel"  
wrote:

On Sun, Nov 27, 2016 at 1:08 PM, Darrell Ball  wrote:

> This patch adds datapaths of interest support where only datapaths of
> local interest are monitored by the ovn-controller ovsdb client.  The
> idea is to do a flood fill in ovn-controller of datapath associations
> calculated by northd. A new column is added to the SB database
> datapath_binding table - related_datapaths to facilitate this so all
> datapaths associations are known quickly in ovn-controller.  This
> allows monitoring to adapt quickly with a single new monitor setting
> for all datapaths of interest locally.
>
> To reduce risk and minimize latency on network churn, only logical
> flows are conditionally monitored for now.  This should provide
> the bulk of the benefit.  This will be re-evaluated later to
> possibly add additional conditional monitoring such as for
> logical ports.
>
> Liran Schour contributed enhancements to the conditional
> monitoring granularity in ovs and also submitted patches
> for ovn usage of conditional monitoring which aided this patch
> though sharing of concepts through code review work.
>
> Ben Pfaff suggested that northd could be used to pre-populate
> related datapaths for ovn-controller to use.  That idea is
> used as part of this patch.
>

I see a few major issues. I don't think this patch in its current state
does what you want it to do. If I understand it correctly, in the
presence of distributed routers it will still populate all flows
everywhere, except for gateway router and corresponding
localnet flows. I have not yet run it manually, but I did play with
the automated tests a little to verify some of my understanding.
See comments inline.

There is an ordering issue wrt build_datapaths() vs build_ports() as you pointed
out below.  I’ll comment more inline at that comment.



>
> CC: Ben Pfaff 
> CC: Liran Schour 
> Signed-off-by: Darrell Ball 
> ---
>
> v7->v8: Start wth logical flow conditional monitoring off.
> Remove deprecated code.
> Recover SB DB version number change.
> Change test to be more definitive.
>
> v6->v7: Rebase
>
> v5->v6: Rebase; fix stale handling.
>
> v4->v5: Correct cleanup of monitors.
> Fix warning.
>
> v3->v4: Refactor after incremental processing backout.
> Limit filtering to logical flows to limit risk.
>
> v2->v3: Line length violation fixups :/
>
> v1->v2: Added logical port removal monitoring handling, factoring
> in recent changes for incremental processing.  Added some
> comments in the code regarding initial conditions for
> database conditional monitoring.
>
>  ovn/controller/binding.c| 10 +++--
>  ovn/controller/lflow.c  |  5 +++
>  ovn/controller/ovn-controller.c | 92 ++
> +++
>  ovn/controller/ovn-controller.h |  3 ++
>  ovn/controller/patch.c  |  6 ++-
>  ovn/northd/ovn-northd.c | 76 ++
>  ovn/ovn-sb.ovsschema| 11 +++--
>  ovn/ovn-sb.xml  |  9 
>  tests/ovn.at|  8 
>  9 files changed, 211 insertions(+), 9 deletions(-)
>
> diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
> index d7b175e..2aff69a 100644
> --- a/ovn/controller/binding.c
> +++ b/ovn/controller/binding.c
> @@ -106,7 +106,8 @@ get_local_iface_ids(const struct ovsrec_bridge 
*br_int,
>
>  static void
>  add_local_datapath(struct hmap *local_datapaths,
> -const struct sbrec_port_binding *binding_rec)
> +const struct sbrec_port_binding *binding_rec,
> +const struct controller_ctx *ctx)
>  {
>  if (get_local_datapath(local_datapaths,
> binding_rec->datapath->tunnel_key)) {
> @@ -118,6 +119,7 @@ add_local_datapath(struct hmap *local_datapaths,
>  memcpy(>uuid, _rec->header_.uuid, sizeof ld->uuid);
>  hmap_insert(local_datapaths, >hmap_node,
>  binding_rec->datapath->tunnel_key);
> +do_datapath_flood_fill(ctx, binding_rec->datapath);
>  }
>
>  static void
> @@ -295,7 +297,7 @@ consider_local_datapath(struct controller_ctx *ctx,
>  /* Add child logical port to the set of all local ports. */
>  sset_add(all_lports, binding_rec->logical_port);
>  }
> -add_local_datapath(local_datapaths, binding_rec);
> +

Re: [ovs-dev] [patch_v8] ovn: Add datapaths of interest filtering.

[Darrell Ball]

On 11/30/16, 3:07 AM, "ovs-dev-boun...@openvswitch.org on behalf of Liran 
Schour"  wrote:

Darrell Ball  wrote on 30/11/2016 02:29:01 AM:

> On Tue, Nov 29, 2016 at 4:10 AM, Liran Schour  wrote:
> Ben Pfaff  wrote on 29/11/2016 12:51:51 AM:
> 
> > I hope that this version is ready to go in.  Liran, are you happy with
> > this version?
> > 
> 
> I did some short evaluation and got the following results:
> Simulate 50 hosts, each host serves 20 logical ports, each logical 
> network is spread over 6 of the 50 hosts.
> 
> Thanks for testing Liran
> 
> Can a few questions be asked so the differences are understood.
> There is expectation that not monitoring logical ports will make a 
> difference, but
> this large is not expected.
> 
> 1) This topology seems to only use logical switches in a flat 
> topology for each tenant;
> is that correct ?

Right. We do not have any logical routers.

> 2)  What is the distribution of logical switches per HV ?

You have 20 logical ports from 20 different logical switches per HV.

Without some incremental processing, port processing which typically should not 
change
that often becomes more important.
Without port conditional monitoring, each HV will monitor 1000 ports vs 20 in 
this test.
I reinstated part of my V3 patch for ports earlier than later since we are in 
between 
incremental processing versions and port conditional monitoring becomes more
important to do earlier, afais.

> 3) What is the total number of logical switches we end up with ?

In that specific test: 167 logical switches.

> 
>  
> 
> 
> Darrell patch:
> --
> Host 1 # flows 1504
> Host 2 # flows 1626
> Host 3 # flows 1443
> ...
> Logical flows = 8016
> Elapsed time 207,990ms
> total of 1002 (in 208ms per port)
> Ports created and bound in 5,648,773,788,428 cycles
> 1% south 69,936,025,644
> 1% north 77,863,484,163
> 97% clients 5,500,974,278,621
> 
> Conditional monitor V16 patch:
> --
> Host 1 # flows 1358
> Host 2 # flows 1482
> Host 3 # flows 1296
> ...
> Logical flows = 8016
> Elapsed time 207,998ms
> total of 1002 (in 208ms per port)
> Ports created and bound in 1,216,848,309,516 cycles
> 
>  
> 
> 4) This is only the CPU time to initially create and bind logical 
> ports to each HV ?
>  How exactly is this time measured ?

You can ignore the time parameter, it is a trace from the past of the 
evaluation system.

My main question was - what is the defined start and end points ?
i.e. when do we start measuring and when do we stop measuring ?

Some of these numbers are really hard to explain.
Hence, I added a separate simple unit test to simply track number of events 
(port and flow) seen by the HVs with conditional monitoring. This is also
easier to interpret and straightforward.


> 5) This total cycles is in the trillions 
> The previous tests from V10 and V15 below seem consistent and in the 2 
digit
> billions, which is at least 30 times less than now.
> Is this same test as before using the 50 hosts configuration  ?
> 

The number that is shown in the tables below is total number of cycles of 
the SB ovsdb-server not overall cycles of all the DC. And as you can see 
(2% south 30,653,838,360) is still in the 2 digit billions.

I see, ok.
Most of the proportional benefit will be on the HV clients anyways.

> Evaluation on simulated environment of 50 hosts and 1000 logical ports 
shows
> the following results (cycles #):
> LN spread over # hosts|master| patch| change
> -
> 1 | 24597200127  | 24339235374  |  1.0%
> 6 | 23788521572  | 19145229352  | 19.5%
>12 | 23886405758  | 17913143176  | 25.0%
>18 | 25812686279  | 23675094540  |  8.2%
>24 | 28414671499  | 24770202308  | 12.8%
>30 | 31487218890  | 28397543436  |  9.8%
>36 | 36116993930  | 34105388739  |  5.5%
>42 | 37898342465  | 38647139083  | -1.9%
>48 | 41637996229  | 41846616306  | -0.5%
>50 | 41679995357  | 43455565977  | -4.2%
> 
> and from V15
> 
> Evaluation on simulated environment of 50 hosts and 1000 logical ports 
shows
> the following results (cycles #):
> 
> LN spread over # hosts|master| patch| change
> 

Re: [ovs-dev] [patch_v8] ovn: Add datapaths of interest filtering.

[Liran Schour]

Darrell Ball  wrote on 30/11/2016 02:29:01 AM:

> On Tue, Nov 29, 2016 at 4:10 AM, Liran Schour  wrote:
> Ben Pfaff  wrote on 29/11/2016 12:51:51 AM:
> 
> > I hope that this version is ready to go in.  Liran, are you happy with
> > this version?
> > 
> 
> I did some short evaluation and got the following results:
> Simulate 50 hosts, each host serves 20 logical ports, each logical 
> network is spread over 6 of the 50 hosts.
> 
> Thanks for testing Liran
> 
> Can a few questions be asked so the differences are understood.
> There is expectation that not monitoring logical ports will make a 
> difference, but
> this large is not expected.
> 
> 1) This topology seems to only use logical switches in a flat 
> topology for each tenant;
> is that correct ?

Right. We do not have any logical routers.

> 2)  What is the distribution of logical switches per HV ?

You have 20 logical ports from 20 different logical switches per HV.

> 3) What is the total number of logical switches we end up with ?

In that specific test: 167 logical switches.

> 
>  
> 
> 
> Darrell patch:
> --
> Host 1 # flows 1504
> Host 2 # flows 1626
> Host 3 # flows 1443
> ...
> Logical flows = 8016
> Elapsed time 207,990ms
> total of 1002 (in 208ms per port)
> Ports created and bound in 5,648,773,788,428 cycles
> 1% south 69,936,025,644
> 1% north 77,863,484,163
> 97% clients 5,500,974,278,621
> 
> Conditional monitor V16 patch:
> --
> Host 1 # flows 1358
> Host 2 # flows 1482
> Host 3 # flows 1296
> ...
> Logical flows = 8016
> Elapsed time 207,998ms
> total of 1002 (in 208ms per port)
> Ports created and bound in 1,216,848,309,516 cycles
> 
>  
> 
> 4) This is only the CPU time to initially create and bind logical 
> ports to each HV ?
>  How exactly is this time measured ?

You can ignore the time parameter, it is a trace from the past of the 
evaluation system.

> 5) This total cycles is in the trillions 
> The previous tests from V10 and V15 below seem consistent and in the 2 
digit
> billions, which is at least 30 times less than now.
> Is this same test as before using the 50 hosts configuration  ?
> 

The number that is shown in the tables below is total number of cycles of 
the SB ovsdb-server not overall cycles of all the DC. And as you can see 
(2% south 30,653,838,360) is still in the 2 digit billions.

> Evaluation on simulated environment of 50 hosts and 1000 logical ports 
shows
> the following results (cycles #):
> LN spread over # hosts|master| patch| change
> -
> 1 | 24597200127  | 24339235374  |  1.0%
> 6 | 23788521572  | 19145229352  | 19.5%
>12 | 23886405758  | 17913143176  | 25.0%
>18 | 25812686279  | 23675094540  |  8.2%
>24 | 28414671499  | 24770202308  | 12.8%
>30 | 31487218890  | 28397543436  |  9.8%
>36 | 36116993930  | 34105388739  |  5.5%
>42 | 37898342465  | 38647139083  | -1.9%
>48 | 41637996229  | 41846616306  | -0.5%
>50 | 41679995357  | 43455565977  | -4.2%
> 
> and from V15
> 
> Evaluation on simulated environment of 50 hosts and 1000 logical ports 
shows
> the following results (cycles #):
> 
> LN spread over # hosts|master| patch| change
> -
> 1 | 24597200127  | 24339235374  |  1.0%
> 6 | 23788521572  | 19145229352  | 19.5%
>12 | 23886405758  | 17913143176  | 25.0%
>18 | 25812686279  | 23675094540  |  8.2%
>24 | 28414671499  | 24770202308  | 12.8%
>30 | 31487218890  | 28397543436  |  9.8%
>36 | 36116993930  | 34105388739  |  5.5%
>42 | 37898342465  | 38647139083  | -1.9%
>48 | 41637996229  | 41846616306  | -0.5%
>50 | 41679995357  | 43455565977  | -4.2%
> 
>  
> 
> 2% south 30,653,838,360
>  
> 
> 3% north 42,543,336,355
> 
> 6) Is this ovsdb NB database server CPU cycles ?; (it is hard to see
> why the cycles
> would differ here, although the numbers are small compared to the HV)

Yes it is. I do not know exactly why you have difference here also.

>  
> 
> 93% clients 1,143,651,134,801
> 
> You can see that this patch significantly degrades the overhead of 
> computation comparing to my origin patch (rebased version).
> 
> An obvious difference is that Darrell's patch has conditions only on
> the Logical_Flow table and the origin patch did that on 
> Port_Binding, Logical_Flow, Multicast_Group and MAC_Binding tables.
> 
>  
> 
> 
> I can resubmit my rebased patch again and Darell can base his 
> changes on top of it or just investigate the problem.
> 
> The code 

Re: [ovs-dev] [patch_v8] ovn: Add datapaths of interest filtering.

[Ben Pfaff]

I hope that this version is ready to go in.  Liran, are you happy with
this version?

On Sun, Nov 27, 2016 at 01:08:59PM -0800, Darrell Ball wrote:
> This patch adds datapaths of interest support where only datapaths of
> local interest are monitored by the ovn-controller ovsdb client.  The
> idea is to do a flood fill in ovn-controller of datapath associations
> calculated by northd. A new column is added to the SB database
> datapath_binding table - related_datapaths to facilitate this so all
> datapaths associations are known quickly in ovn-controller.  This
> allows monitoring to adapt quickly with a single new monitor setting
> for all datapaths of interest locally.
> 
> To reduce risk and minimize latency on network churn, only logical
> flows are conditionally monitored for now.  This should provide
> the bulk of the benefit.  This will be re-evaluated later to
> possibly add additional conditional monitoring such as for
> logical ports.
> 
> Liran Schour contributed enhancements to the conditional
> monitoring granularity in ovs and also submitted patches
> for ovn usage of conditional monitoring which aided this patch
> though sharing of concepts through code review work.
> 
> Ben Pfaff suggested that northd could be used to pre-populate
> related datapaths for ovn-controller to use.  That idea is
> used as part of this patch.
> 
> CC: Ben Pfaff 
> CC: Liran Schour 
> Signed-off-by: Darrell Ball 
> ---
> 
> v7->v8: Start wth logical flow conditional monitoring off.
> Remove deprecated code.
> Recover SB DB version number change.
> Change test to be more definitive. 
> 
> v6->v7: Rebase
> 
> v5->v6: Rebase; fix stale handling.
> 
> v4->v5: Correct cleanup of monitors.
> Fix warning.
> 
> v3->v4: Refactor after incremental processing backout.
> Limit filtering to logical flows to limit risk.
> 
> v2->v3: Line length violation fixups :/
> 
> v1->v2: Added logical port removal monitoring handling, factoring
> in recent changes for incremental processing.  Added some
> comments in the code regarding initial conditions for
> database conditional monitoring.
> 
>  ovn/controller/binding.c| 10 +++--
>  ovn/controller/lflow.c  |  5 +++
>  ovn/controller/ovn-controller.c | 92 
> +
>  ovn/controller/ovn-controller.h |  3 ++
>  ovn/controller/patch.c  |  6 ++-
>  ovn/northd/ovn-northd.c | 76 ++
>  ovn/ovn-sb.ovsschema| 11 +++--
>  ovn/ovn-sb.xml  |  9 
>  tests/ovn.at|  8 
>  9 files changed, 211 insertions(+), 9 deletions(-)
> 
> diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
> index d7b175e..2aff69a 100644
> --- a/ovn/controller/binding.c
> +++ b/ovn/controller/binding.c
> @@ -106,7 +106,8 @@ get_local_iface_ids(const struct ovsrec_bridge *br_int,
>  
>  static void
>  add_local_datapath(struct hmap *local_datapaths,
> -const struct sbrec_port_binding *binding_rec)
> +const struct sbrec_port_binding *binding_rec,
> +const struct controller_ctx *ctx)
>  {
>  if (get_local_datapath(local_datapaths,
> binding_rec->datapath->tunnel_key)) {
> @@ -118,6 +119,7 @@ add_local_datapath(struct hmap *local_datapaths,
>  memcpy(>uuid, _rec->header_.uuid, sizeof ld->uuid);
>  hmap_insert(local_datapaths, >hmap_node,
>  binding_rec->datapath->tunnel_key);
> +do_datapath_flood_fill(ctx, binding_rec->datapath);
>  }
>  
>  static void
> @@ -295,7 +297,7 @@ consider_local_datapath(struct controller_ctx *ctx,
>  /* Add child logical port to the set of all local ports. */
>  sset_add(all_lports, binding_rec->logical_port);
>  }
> -add_local_datapath(local_datapaths, binding_rec);
> +add_local_datapath(local_datapaths, binding_rec, ctx);
>  if (iface_rec && qos_map && ctx->ovs_idl_txn) {
>  get_qos_params(binding_rec, qos_map);
>  }
> @@ -330,7 +332,7 @@ consider_local_datapath(struct controller_ctx *ctx,
>  }
>  
>  sset_add(all_lports, binding_rec->logical_port);
> -add_local_datapath(local_datapaths, binding_rec);
> +add_local_datapath(local_datapaths, binding_rec, ctx);
>  if (binding_rec->chassis == chassis_rec) {
>  return;
>  }
> @@ -344,7 +346,7 @@ consider_local_datapath(struct controller_ctx *ctx,
>  const char *chassis = smap_get(_rec->options,
> "l3gateway-chassis");
>  if (!strcmp(chassis, chassis_rec->name) && ctx->ovnsb_idl_txn) {
> -add_local_datapath(local_datapaths, binding_rec);
> +add_local_datapath(local_datapaths, binding_rec, ctx);
>  }
>  } else if (chassis_rec && binding_rec->chassis == chassis_rec) {
>