[ovs-dev] [PATCH v4] ovn-controller: add quiet mode

2016-10-06 Thread Ryan Moats 
As discussed in [1], what the incremental processing code
actually accomplished was that the ovn-controller would
be "quiet" and not burn CPU when things weren't changing.
This patch set recreates this state by calculating whether
changes have occured that would require a full calculation
to be performed.  It does this by persisting a copy of
the localvif_to_ofport and tunnel information in the
controller module, rather than in the physical.c module
as was the case with previous commits.

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/ofctrl.c |   2 +
 ovn/controller/ovn-controller.c |  67 +++
 ovn/controller/ovn-controller.h |   1 +
 ovn/controller/patch.c  |   5 ++
 ovn/controller/physical.c   | 100 
 ovn/controller/physical.h   |   5 ++
 6 files changed, 121 insertions(+), 59 deletions(-)

diff --git a/ovn/controller/ofctrl.c b/ovn/controller/ofctrl.c
index 1d8fbf3..a3f465f 100644
--- a/ovn/controller/ofctrl.c
+++ b/ovn/controller/ofctrl.c
@@ -377,6 +377,8 @@ run_S_CLEAR_FLOWS(void)
 queue_msg(encode_group_mod());
 ofputil_uninit_group_mod();
 
+force_full_process();
+
 /* Clear existing groups, to match the state of the switch. */
 if (groups) {
 ovn_group_table_clear(groups, true);
diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c
index 4ac1425..9762683 100644
--- a/ovn/controller/ovn-controller.c
+++ b/ovn/controller/ovn-controller.c
@@ -397,6 +397,23 @@ get_nb_cfg(struct ovsdb_idl *idl)
 return sb ? sb->nb_cfg : 0;
 }
 
+/* Contains mapping of localvifs to OF ports for detecting if the physical
+ * configuration of the switch has changed. */
+static struct simap localvif_to_ofport =
+SIMAP_INITIALIZER(_to_ofport);
+
+/* Contains the list of known tunnels. */
+static struct hmap tunnels = HMAP_INITIALIZER();
+
+/* The last sequence number seen from the southbound IDL. */
+static unsigned int seqno = 0;
+
+void
+force_full_process(void)
+{
+seqno = 0;
+}
+
 int
 main(int argc, char *argv[])
 {
@@ -523,36 +540,47 @@ main(int argc, char *argv[])
 _lports);
 }
 
+bool physical_change = true;
 if (br_int && chassis) {
+enum mf_field_id mff_ovn_geneve = ofctrl_run(br_int,
+ _ct_zones);
+physical_change = detect_and_save_physical_changes(
+_to_ofport, , mff_ovn_geneve, br_int,
+chassis_id);
+unsigned int cur_seqno = ovsdb_idl_get_seqno(ovnsb_idl_loop.idl);
+
 patch_run(, br_int, chassis_id, _datapaths,
   _datapaths);
 
 static struct lport_index lports;
-static struct mcgroup_index mcgroups;
 lport_index_init(, ctx.ovnsb_idl);
-mcgroup_index_init(, ctx.ovnsb_idl);
-
-enum mf_field_id mff_ovn_geneve = ofctrl_run(br_int,
- _ct_zones);
 
 pinctrl_run(, , br_int, chassis_id, _datapaths);
 update_ct_zones(_lports, _datapaths, _zones,
 ct_zone_bitmap, _ct_zones);
 if (ctx.ovs_idl_txn) {
-commit_ct_zones(br_int, _ct_zones);
-
-struct hmap flow_table = HMAP_INITIALIZER(_table);
-lflow_run(, , , _datapaths,
-  _datapaths, _table, _zones,
-  _table);
-
-physical_run(, mff_ovn_geneve,
- br_int, chassis_id, _zones, _table,
- _datapaths, _datapaths);
-
-ofctrl_put(_table, _ct_zones,
-   get_nb_cfg(ctx.ovnsb_idl));
-hmap_destroy(_table);
+if (physical_change || seqno < cur_seqno) {
+seqno = cur_seqno;
+
+commit_ct_zones(br_int, _ct_zones);
+
+static struct mcgroup_index mcgroups;
+mcgroup_index_init(, ctx.ovnsb_idl);
+
+struct hmap flow_table = HMAP_INITIALIZER(_table);
+lflow_run(, , , _datapaths,
+  _datapaths, _table, _zones,
+  _table);
+
+physical_run(, mff_ovn_geneve,
+ br_int, chassis_id, _zones, _table,
+ _datapaths, _datapaths);
+
+ofctrl_put(_table, _ct_zones,
+   get_nb_cfg(ctx.ovnsb_idl));
+hmap_destroy(_table);
+mcgroup_index_destroy();
+}
 if (ctx.ovnsb_idl_txn) {
 int64_t cur_cfg = ofctrl_get_cur_cfg();
 if

Re: [ovs-dev] [ovs-dev,v2,2/4] ovn-controller: add quiet mode

2016-10-05 Thread Ryan Moats 


"dev" <dev-boun...@openvswitch.org> wrote on 10/05/2016 01:19:34 PM:

> From: Ryan Moats/Omaha/IBM@IBMUS
> To: Ben Pfaff <b...@ovn.org>
> Cc: dev@openvswitch.org
> Date: 10/05/2016 01:20 PM
> Subject: Re: [ovs-dev] [ovs-dev,v2,2/4] ovn-controller: add quiet mode
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> Ben Pfaff <b...@ovn.org> wrote on 10/05/2016 01:04:36 PM:
>
> > From: Ben Pfaff <b...@ovn.org>
> > To: Ryan Moats/Omaha/IBM@IBMUS
> > Cc: dev@openvswitch.org
> > Date: 10/05/2016 01:04 PM
> > Subject: Re: [ovs-dev,v2,2/4] ovn-controller: add quiet mode
> >
> > On Wed, Oct 05, 2016 at 12:52:57PM -0500, Ryan Moats wrote:
> > > Ben Pfaff <b...@ovn.org> wrote on 10/05/2016 12:37:26 PM:
> > >
> > > > From: Ben Pfaff <b...@ovn.org>
> > > > To: Ryan Moats/Omaha/IBM@IBMUS
> > > > Cc: dev@openvswitch.org
> > > > Date: 10/05/2016 12:37 PM
> > > > Subject: Re: [ovs-dev,v2,2/4] ovn-controller: add quiet mode
> > > >
> > > > On Tue, Oct 04, 2016 at 05:11:37PM -0500, Ryan Moats wrote:
> > > > > Ben Pfaff <b...@ovn.org> wrote on 10/04/2016 12:14:32 PM:
> > > > >
> > > > > > From: Ben Pfaff <b...@ovn.org>
> > > > > > To: Ryan Moats/Omaha/IBM@IBMUS
> > > > > > Cc: dev@openvswitch.org
> > > > > > Date: 10/04/2016 12:14 PM
> > > > > > Subject: Re: [ovs-dev,v2,2/4] ovn-controller: add quiet mode
> > > > > >
> > > > > > On Wed, Aug 31, 2016 at 03:22:44PM +, Ryan Moats wrote:
> > > > > > > As discussed in [1], what the incremental processing code
> > > > > > > actually accomplished was that the ovn-controller would
> > > > > > > be "quiet" and not burn CPU when things weren't changing.
> > > > > > > This patch set recreates this state by calculating whether
> > > > > > > changes have occured that would require a full calculation
> > > > > > > to be performed.  It does this by persisting a copy of
> > > > > > > the localvif_to_ofport and tunnel information in the
> > > > > > > controller module, rather than in the physical.c module
> > > > > > > as was the case with previous commits.
> > > > > > >
> > > > > > > [1]
> http://openvswitch.org/pipermail/dev/2016-August/078272.html
> > > > > > >
> > > > > > > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> > > > > >
> > > > > > Hi Ryan.
> > > > > >
> > > > > > I like the idea behind this patch.  However, it no longer
applies
> to
> > > > > > master, so it needs a rebase.
> > > > >
> > > > > So done, but before submitting a new patch
> > > > >
> > > > > >
> > > > > > It also seems like this TODO should be addressed:
> > > > > > +/* TODO (regXboi): this next line is needed for the 3
> HVs, 3
> > > LS,
> > > > > > + * 3 lports/LS, 1 LR test case, but has the potential
> side
> > > > > effect
> > > > > > + * of defeating quiet mode once a logical router leads
> to
> > > > > creating
> > > > > > + * patch ports. Need to understand the failure mode
> better
> > > and
> > > > > > + * what is needed to remove this. */
> > > > > > +force_full_process();
> > > > >
> > > > > I've been looking at what happens here and I'm seeing some
> signatures
> > > > > that concern me.  The test case that fails is no longer the cited
> one
> > > above
> > > > > but is "2 HVs, 4 lports/HV, localnet ports" ...
> > > > >
> > > > > What I'm seeing when I peer in is that the information populating
> the
> > > > > local_datapath structures doesn't appear to be consistent on each
> pass
> > > > > through binding_run: Looking at the ovn-controller process under
> hv2
> > > > > for the above test (when it passes), I'll see signatures that
look
> > > > > like the following:
> > > > >
> > > > > 2016-10-04T21:57:58.257Z|00020|physical|INFO|looking at binding
> record
> > > > > 3736404b-c69d-4878-8d45-81ad9be0

[ovs-dev] [PATCH] ovn-controller: honor ovs_idl_txn when calculating and installing flows.

2016-10-05 Thread Ryan Moats 
ovs_idl_txn is checked before various routines (like patch_run) execute.
However, flow calculation and installation does not also check this
variable, which can lead to oscillations as described in [1].

[1] http://openvswitch.org/pipermail/dev/2016-October/080247.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
Note: This patch also applies to the 2.6 branch

 ovn/controller/ovn-controller.c | 45 +++--
 1 file changed, 21 insertions(+), 24 deletions(-)

diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c
index 00392ca..4ac1425 100644
--- a/ovn/controller/ovn-controller.c
+++ b/ovn/controller/ovn-controller.c
@@ -312,14 +312,9 @@ update_ct_zones(struct sset *lports, struct hmap 
*patched_datapaths,
 }
 
 static void
-commit_ct_zones(struct controller_ctx *ctx,
-const struct ovsrec_bridge *br_int,
+commit_ct_zones(const struct ovsrec_bridge *br_int,
 struct shash *pending_ct_zones)
 {
-if (!ctx->ovs_idl_txn) {
-return;
-}
-
 struct smap new_ids;
 smap_clone(_ids, _int->external_ids);
 
@@ -543,24 +538,26 @@ main(int argc, char *argv[])
 pinctrl_run(, , br_int, chassis_id, _datapaths);
 update_ct_zones(_lports, _datapaths, _zones,
 ct_zone_bitmap, _ct_zones);
-commit_ct_zones(, br_int, _ct_zones);
-
-struct hmap flow_table = HMAP_INITIALIZER(_table);
-lflow_run(, , , _datapaths,
-  _datapaths, _table, _zones,
-  _table);
-
-physical_run(, mff_ovn_geneve,
- br_int, chassis_id, _zones, _table,
- _datapaths, _datapaths);
-
-ofctrl_put(_table, _ct_zones,
-   get_nb_cfg(ctx.ovnsb_idl));
-hmap_destroy(_table);
-if (ctx.ovnsb_idl_txn) {
-int64_t cur_cfg = ofctrl_get_cur_cfg();
-if (cur_cfg && cur_cfg != chassis->nb_cfg) {
-sbrec_chassis_set_nb_cfg(chassis, cur_cfg);
+if (ctx.ovs_idl_txn) {
+commit_ct_zones(br_int, _ct_zones);
+
+struct hmap flow_table = HMAP_INITIALIZER(_table);
+lflow_run(, , , _datapaths,
+  _datapaths, _table, _zones,
+  _table);
+
+physical_run(, mff_ovn_geneve,
+ br_int, chassis_id, _zones, _table,
+ _datapaths, _datapaths);
+
+ofctrl_put(_table, _ct_zones,
+   get_nb_cfg(ctx.ovnsb_idl));
+hmap_destroy(_table);
+if (ctx.ovnsb_idl_txn) {
+int64_t cur_cfg = ofctrl_get_cur_cfg();
+if (cur_cfg && cur_cfg != chassis->nb_cfg) {
+sbrec_chassis_set_nb_cfg(chassis, cur_cfg);
+}
 }
 }
 mcgroup_index_destroy();
-- 
2.7.4

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [ovs-dev,v2,2/4] ovn-controller: add quiet mode

2016-10-05 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 10/05/2016 01:04:36 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 10/05/2016 01:04 PM
> Subject: Re: [ovs-dev,v2,2/4] ovn-controller: add quiet mode
>
> On Wed, Oct 05, 2016 at 12:52:57PM -0500, Ryan Moats wrote:
> > Ben Pfaff <b...@ovn.org> wrote on 10/05/2016 12:37:26 PM:
> >
> > > From: Ben Pfaff <b...@ovn.org>
> > > To: Ryan Moats/Omaha/IBM@IBMUS
> > > Cc: dev@openvswitch.org
> > > Date: 10/05/2016 12:37 PM
> > > Subject: Re: [ovs-dev,v2,2/4] ovn-controller: add quiet mode
> > >
> > > On Tue, Oct 04, 2016 at 05:11:37PM -0500, Ryan Moats wrote:
> > > > Ben Pfaff <b...@ovn.org> wrote on 10/04/2016 12:14:32 PM:
> > > >
> > > > > From: Ben Pfaff <b...@ovn.org>
> > > > > To: Ryan Moats/Omaha/IBM@IBMUS
> > > > > Cc: dev@openvswitch.org
> > > > > Date: 10/04/2016 12:14 PM
> > > > > Subject: Re: [ovs-dev,v2,2/4] ovn-controller: add quiet mode
> > > > >
> > > > > On Wed, Aug 31, 2016 at 03:22:44PM +, Ryan Moats wrote:
> > > > > > As discussed in [1], what the incremental processing code
> > > > > > actually accomplished was that the ovn-controller would
> > > > > > be "quiet" and not burn CPU when things weren't changing.
> > > > > > This patch set recreates this state by calculating whether
> > > > > > changes have occured that would require a full calculation
> > > > > > to be performed.  It does this by persisting a copy of
> > > > > > the localvif_to_ofport and tunnel information in the
> > > > > > controller module, rather than in the physical.c module
> > > > > > as was the case with previous commits.
> > > > > >
> > > > > > [1]
http://openvswitch.org/pipermail/dev/2016-August/078272.html
> > > > > >
> > > > > > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> > > > >
> > > > > Hi Ryan.
> > > > >
> > > > > I like the idea behind this patch.  However, it no longer applies
to
> > > > > master, so it needs a rebase.
> > > >
> > > > So done, but before submitting a new patch
> > > >
> > > > >
> > > > > It also seems like this TODO should be addressed:
> > > > > +/* TODO (regXboi): this next line is needed for the 3
HVs, 3
> > LS,
> > > > > + * 3 lports/LS, 1 LR test case, but has the potential
side
> > > > effect
> > > > > + * of defeating quiet mode once a logical router leads
to
> > > > creating
> > > > > + * patch ports. Need to understand the failure mode
better
> > and
> > > > > + * what is needed to remove this. */
> > > > > +force_full_process();
> > > >
> > > > I've been looking at what happens here and I'm seeing some
signatures
> > > > that concern me.  The test case that fails is no longer the cited
one
> > above
> > > > but is "2 HVs, 4 lports/HV, localnet ports" ...
> > > >
> > > > What I'm seeing when I peer in is that the information populating
the
> > > > local_datapath structures doesn't appear to be consistent on each
pass
> > > > through binding_run: Looking at the ovn-controller process under
hv2
> > > > for the above test (when it passes), I'll see signatures that look
> > > > like the following:
> > > >
> > > > 2016-10-04T21:57:58.257Z|00020|physical|INFO|looking at binding
record
> > > > 3736404b-c69d-4878-8d45-81ad9be06f5f
> > > > 2016-10-04T21:57:58.257Z|00021|physical|INFO|dp_key is 1
> > > > 2016-10-04T21:57:58.257Z|00022|physical|INFO|looking for ofport of
lp11
> > > > (LP)
> > > > 2016-10-04T21:57:58.257Z|00023|physical|INFO|looking for ofport of
ln1
> > > > (localnet)
> > > > ...
> > > > 2016-10-04T21:57:58.259Z|00034|physical|INFO|looking at binding
record
> > > > 3736404b-c69d-4878-8d45-81ad9be06f5f
> > > > 2016-10-04T21:57:58.259Z|00035|physical|INFO|dp_key is 1
> > > > 2016-10-04T21:57:58.259Z|00036|physical|INFO|looking for ofport of
lp11
> > > > (LP)
> > > > 2016-10-04T21:57:58.259Z|00037|physical|INFO|looking for tunnel to
hv1
> > > >

Re: [ovs-dev] [ovs-dev,v2,2/4] ovn-controller: add quiet mode

2016-10-05 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 10/05/2016 12:37:26 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 10/05/2016 12:37 PM
> Subject: Re: [ovs-dev,v2,2/4] ovn-controller: add quiet mode
>
> On Tue, Oct 04, 2016 at 05:11:37PM -0500, Ryan Moats wrote:
> > Ben Pfaff <b...@ovn.org> wrote on 10/04/2016 12:14:32 PM:
> >
> > > From: Ben Pfaff <b...@ovn.org>
> > > To: Ryan Moats/Omaha/IBM@IBMUS
> > > Cc: dev@openvswitch.org
> > > Date: 10/04/2016 12:14 PM
> > > Subject: Re: [ovs-dev,v2,2/4] ovn-controller: add quiet mode
> > >
> > > On Wed, Aug 31, 2016 at 03:22:44PM +, Ryan Moats wrote:
> > > > As discussed in [1], what the incremental processing code
> > > > actually accomplished was that the ovn-controller would
> > > > be "quiet" and not burn CPU when things weren't changing.
> > > > This patch set recreates this state by calculating whether
> > > > changes have occured that would require a full calculation
> > > > to be performed.  It does this by persisting a copy of
> > > > the localvif_to_ofport and tunnel information in the
> > > > controller module, rather than in the physical.c module
> > > > as was the case with previous commits.
> > > >
> > > > [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
> > > >
> > > > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> > >
> > > Hi Ryan.
> > >
> > > I like the idea behind this patch.  However, it no longer applies to
> > > master, so it needs a rebase.
> >
> > So done, but before submitting a new patch
> >
> > >
> > > It also seems like this TODO should be addressed:
> > > +/* TODO (regXboi): this next line is needed for the 3 HVs, 3
LS,
> > > + * 3 lports/LS, 1 LR test case, but has the potential side
> > effect
> > > + * of defeating quiet mode once a logical router leads to
> > creating
> > > + * patch ports. Need to understand the failure mode better
and
> > > + * what is needed to remove this. */
> > > +force_full_process();
> >
> > I've been looking at what happens here and I'm seeing some signatures
> > that concern me.  The test case that fails is no longer the cited one
above
> > but is "2 HVs, 4 lports/HV, localnet ports" ...
> >
> > What I'm seeing when I peer in is that the information populating the
> > local_datapath structures doesn't appear to be consistent on each pass
> > through binding_run: Looking at the ovn-controller process under hv2
> > for the above test (when it passes), I'll see signatures that look
> > like the following:
> >
> > 2016-10-04T21:57:58.257Z|00020|physical|INFO|looking at binding record
> > 3736404b-c69d-4878-8d45-81ad9be06f5f
> > 2016-10-04T21:57:58.257Z|00021|physical|INFO|dp_key is 1
> > 2016-10-04T21:57:58.257Z|00022|physical|INFO|looking for ofport of lp11
> > (LP)
> > 2016-10-04T21:57:58.257Z|00023|physical|INFO|looking for ofport of ln1
> > (localnet)
> > ...
> > 2016-10-04T21:57:58.259Z|00034|physical|INFO|looking at binding record
> > 3736404b-c69d-4878-8d45-81ad9be06f5f
> > 2016-10-04T21:57:58.259Z|00035|physical|INFO|dp_key is 1
> > 2016-10-04T21:57:58.259Z|00036|physical|INFO|looking for ofport of lp11
> > (LP)
> > 2016-10-04T21:57:58.259Z|00037|physical|INFO|looking for tunnel to hv1
> > ...
>
> This sort of oscillation seems super-weird to me.  I'd also like to
> learn more.
>

I found the problem and fixed it in the v3 patch set - some of the methods
(patch_run for example) now include a test of ovs_idl_txn before they run
and some don't (physical_run for example), so you can see guess what was
happening...

(If we had a txn to work with everything is fine, but without one,
physical_run was trying to calculate flows based on local datapaths
with no localnet ports being defined, which leads to the above
oscillation.)

I added a txn check to the logic for whether to call things or not in
ovn-controller.c and I added a note in the email about thinking that a
follow-on patch to clean up all of the ovs_idl_txn gates might make sense
(so that nobody trips on this in the future)

Ryan
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v3] ovn-controller: add quiet mode

2016-10-05 Thread Ryan Moats 
As discussed in [1], what the incremental processing code
actually accomplished was that the ovn-controller would
be "quiet" and not burn CPU when things weren't changing.
This patch set recreates this state by calculating whether
changes have occured that would require a full calculation
to be performed.  It does this by persisting a copy of
the localvif_to_ofport and tunnel information in the
controller module, rather than in the physical.c module
as was the case with previous commits.

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
Note: it may make sense to consider a follow on patch to clean
up all the tests for the existence of ovs_idl_txn into the
main ovn-controller loop...

 ovn/controller/ofctrl.c |   2 +
 ovn/controller/ovn-controller.c |  71 +++-
 ovn/controller/ovn-controller.h |   1 +
 ovn/controller/patch.c  |   5 ++
 ovn/controller/physical.c   | 100 
 ovn/controller/physical.h   |   5 ++
 6 files changed, 123 insertions(+), 61 deletions(-)

diff --git a/ovn/controller/ofctrl.c b/ovn/controller/ofctrl.c
index 1d8fbf3..a3f465f 100644
--- a/ovn/controller/ofctrl.c
+++ b/ovn/controller/ofctrl.c
@@ -377,6 +377,8 @@ run_S_CLEAR_FLOWS(void)
 queue_msg(encode_group_mod());
 ofputil_uninit_group_mod();
 
+force_full_process();
+
 /* Clear existing groups, to match the state of the switch. */
 if (groups) {
 ovn_group_table_clear(groups, true);
diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c
index 00392ca..3c6b8b3 100644
--- a/ovn/controller/ovn-controller.c
+++ b/ovn/controller/ovn-controller.c
@@ -402,6 +402,23 @@ get_nb_cfg(struct ovsdb_idl *idl)
 return sb ? sb->nb_cfg : 0;
 }
 
+/* Contains mapping of localvifs to OF ports for detecting if the physical
+ * configuration of the switch has changed. */
+static struct simap localvif_to_ofport =
+SIMAP_INITIALIZER(_to_ofport);
+
+/* Contains the list of known tunnels. */
+static struct hmap tunnels = HMAP_INITIALIZER();
+
+/* The last sequence number seen from the southbound IDL. */
+static unsigned int seqno = 0;
+
+void
+force_full_process(void)
+{
+seqno = 0;
+}
+
 int
 main(int argc, char *argv[])
 {
@@ -528,42 +545,52 @@ main(int argc, char *argv[])
 _lports);
 }
 
+bool physical_change = true;
 if (br_int && chassis) {
+enum mf_field_id mff_ovn_geneve = ofctrl_run(br_int,
+ _ct_zones);
+physical_change = detect_and_save_physical_changes(
+_to_ofport, , mff_ovn_geneve, br_int,
+chassis_id);
+unsigned int cur_seqno = ovsdb_idl_get_seqno(ovnsb_idl_loop.idl);
+
 patch_run(, br_int, chassis_id, _datapaths,
   _datapaths);
 
 static struct lport_index lports;
-static struct mcgroup_index mcgroups;
 lport_index_init(, ctx.ovnsb_idl);
-mcgroup_index_init(, ctx.ovnsb_idl);
-
-enum mf_field_id mff_ovn_geneve = ofctrl_run(br_int,
- _ct_zones);
 
 pinctrl_run(, , br_int, chassis_id, _datapaths);
 update_ct_zones(_lports, _datapaths, _zones,
 ct_zone_bitmap, _ct_zones);
 commit_ct_zones(, br_int, _ct_zones);
 
-struct hmap flow_table = HMAP_INITIALIZER(_table);
-lflow_run(, , , _datapaths,
-  _datapaths, _table, _zones,
-  _table);
-
-physical_run(, mff_ovn_geneve,
- br_int, chassis_id, _zones, _table,
- _datapaths, _datapaths);
-
-ofctrl_put(_table, _ct_zones,
-   get_nb_cfg(ctx.ovnsb_idl));
-hmap_destroy(_table);
-if (ctx.ovnsb_idl_txn) {
-int64_t cur_cfg = ofctrl_get_cur_cfg();
-if (cur_cfg && cur_cfg != chassis->nb_cfg) {
-sbrec_chassis_set_nb_cfg(chassis, cur_cfg);
+if (ctx.ovs_idl_txn && (physical_change || seqno < cur_seqno)) {
+seqno = cur_seqno;
+
+static struct mcgroup_index mcgroups;
+mcgroup_index_init(, ctx.ovnsb_idl);
+
+struct hmap flow_table = HMAP_INITIALIZER(_table);
+lflow_run(, , , _datapaths,
+  _datapaths, _table, _zones,
+  _table);
+
+physical_run(, mff_ovn_geneve,
+ br_int, chassis_id, _zones, _table,
+ _datapaths, _datapaths);
+
+ofctrl_put(_table, _ct_zones,
+   get_nb_cfg(ctx.

Re: [ovs-dev] [ovs-dev,v2,2/4] ovn-controller: add quiet mode

2016-10-04 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 10/04/2016 12:14:32 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 10/04/2016 12:14 PM
> Subject: Re: [ovs-dev,v2,2/4] ovn-controller: add quiet mode
>
> On Wed, Aug 31, 2016 at 03:22:44PM +, Ryan Moats wrote:
> > As discussed in [1], what the incremental processing code
> > actually accomplished was that the ovn-controller would
> > be "quiet" and not burn CPU when things weren't changing.
> > This patch set recreates this state by calculating whether
> > changes have occured that would require a full calculation
> > to be performed.  It does this by persisting a copy of
> > the localvif_to_ofport and tunnel information in the
> > controller module, rather than in the physical.c module
> > as was the case with previous commits.
> >
> > [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
> >
> > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
>
> Hi Ryan.
>
> I like the idea behind this patch.  However, it no longer applies to
> master, so it needs a rebase.

So done, but before submitting a new patch

>
> It also seems like this TODO should be addressed:
> +/* TODO (regXboi): this next line is needed for the 3 HVs, 3 LS,
> + * 3 lports/LS, 1 LR test case, but has the potential side
effect
> + * of defeating quiet mode once a logical router leads to
creating
> + * patch ports. Need to understand the failure mode better and
> + * what is needed to remove this. */
> +force_full_process();

I've been looking at what happens here and I'm seeing some signatures
that concern me.  The test case that fails is no longer the cited one above
but is "2 HVs, 4 lports/HV, localnet ports" ...

What I'm seeing when I peer in is that the information populating the
local_datapath structures doesn't appear to be consistent on each pass
through binding_run: Looking at the ovn-controller process under hv2
for the above test (when it passes), I'll see signatures that look
like the following:

2016-10-04T21:57:58.257Z|00020|physical|INFO|looking at binding record
3736404b-c69d-4878-8d45-81ad9be06f5f
2016-10-04T21:57:58.257Z|00021|physical|INFO|dp_key is 1
2016-10-04T21:57:58.257Z|00022|physical|INFO|looking for ofport of lp11
(LP)
2016-10-04T21:57:58.257Z|00023|physical|INFO|looking for ofport of ln1
(localnet)
...
2016-10-04T21:57:58.259Z|00034|physical|INFO|looking at binding record
3736404b-c69d-4878-8d45-81ad9be06f5f
2016-10-04T21:57:58.259Z|00035|physical|INFO|dp_key is 1
2016-10-04T21:57:58.259Z|00036|physical|INFO|looking for ofport of lp11
(LP)
2016-10-04T21:57:58.259Z|00037|physical|INFO|looking for tunnel to hv1
...
2016-10-04T21:57:58.263Z|00054|physical|INFO|looking at binding record
3736404b-c69d-4878-8d45-81ad9be06f5f
2016-10-04T21:57:58.263Z|00055|physical|INFO|dp_key is 1
2016-10-04T21:57:58.263Z|00056|physical|INFO|looking for ofport of lp11
(LP)
2016-10-04T21:57:58.263Z|00057|physical|INFO|looking for ofport of ln1
(localnet)
...
2016-10-04T21:57:58.275Z|00115|physical|INFO|looking at binding record
3736404b-c69d-4878-8d45-81ad9be06f5f
2016-10-04T21:57:58.275Z|00116|physical|INFO|dp_key is 1
2016-10-04T21:57:58.275Z|00117|physical|INFO|looking for ofport of lp11
(LP)
2016-10-04T21:57:58.275Z|00118|physical|INFO|looking for tunnel to hv1
...
2016-10-04T21:57:58.281Z|00133|physical|INFO|looking at binding record
3736404b-c69d-4878-8d45-81ad9be06f5f
2016-10-04T21:57:58.281Z|00134|physical|INFO|dp_key is 1
2016-10-04T21:57:58.281Z|00135|physical|INFO|looking for ofport of lp11
(LP)
2016-10-04T21:57:58.281Z|00136|physical|INFO|looking for ofport of ln1
(localnet)
...
2016-10-04T21:57:58.296Z|00203|physical|INFO|looking at binding record
3736404b-c69d-4878-8d45-81ad9be06f5f
2016-10-04T21:57:58.296Z|00204|physical|INFO|dp_key is 1
2016-10-04T21:57:58.296Z|00205|physical|INFO|looking for ofport of lp11
(LP)
2016-10-04T21:57:58.296Z|00206|physical|INFO|looking for tunnel to hv1
...
2016-10-04T21:57:58.302Z|00241|physical|INFO|looking at binding record
3736404b-c69d-4878-8d45-81ad9be06f5f
2016-10-04T21:57:58.302Z|00242|physical|INFO|dp_key is 1
2016-10-04T21:57:58.302Z|00243|physical|INFO|looking for ofport of lp11
(LP)
2016-10-04T21:57:58.302Z|00244|physical|INFO|looking for ofport of ln1
(localnet)
...
2016-10-04T21:57:58.327Z|00307|physical|INFO|looking at binding record
3736404b-c69d-4878-8d45-81ad9be06f5f
2016-10-04T21:57:58.327Z|00308|physical|INFO|dp_key is 1
2016-10-04T21:57:58.328Z|00309|physical|INFO|looking for ofport of lp11
(LP)
2016-10-04T21:57:58.328Z|00310|physical|INFO|looking for tunnel to hv1
...
2016-10-04T21:57:58.341Z|00341|physical|INFO|looking at binding record
3736404b-c69d-4878-8d45-81ad9be06f5f
2016-10-04T21:57:58.341Z|00342|physical|INFO|dp_key is 1
2016-10-04T

[ovs-dev] [PATCH] ofproto-dpif-xlate: Fix memory leak in execute_controller_action.

2016-09-20 Thread Ryan Moats 
commit df70a7731 ("ofproto-dpif-xlate: Allow translating
without side-effects.") created a memory leak by removing the
dp_packet_delete statement in execute_controller_action that
freed the earlier cloned packet.  This commit restores this
statement to the end of the method.

This issue is only seen in the master branch.

Fixes: df70a7731 ("ofproto-dpif-xlate: Allow translating without side-effects.")
Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ofproto/ofproto-dpif-xlate.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/ofproto/ofproto-dpif-xlate.c b/ofproto/ofproto-dpif-xlate.c
index f17cb79..71ffca0 100644
--- a/ofproto/ofproto-dpif-xlate.c
+++ b/ofproto/ofproto-dpif-xlate.c
@@ -3648,6 +3648,8 @@ execute_controller_action(struct xlate_ctx *ctx, int len,
 entry->controller.ofproto = ctx->xbridge->ofproto;
 entry->controller.am = am;
 }
+
+dp_packet_delete(packet);
 }
 
 static void
-- 
2.7.0

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH] ovn-nbctl, tests: Clean up noisy memory leaks

2016-09-09 Thread Ryan Moats 
When run with valgrind, ovn-nbctl.c and tests/test-ovn.c reveal
memory leaks of their own.  This patch cleans these up so that
they don't create noise when looking for leaks in the OVN daemon
processes.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/utilities/ovn-nbctl.c | 1 +
 tests/test-ovn.c  | 4 
 2 files changed, 5 insertions(+)

diff --git a/ovn/utilities/ovn-nbctl.c b/ovn/utilities/ovn-nbctl.c
index d6d64ea..a37fe51 100644
--- a/ovn/utilities/ovn-nbctl.c
+++ b/ovn/utilities/ovn-nbctl.c
@@ -1608,6 +1608,7 @@ nbctl_lr_route_add(struct ctl_context *ctx)
 }
 
 if (!may_exist) {
+free(rt_prefix);
 ctl_fatal("duplicate prefix: %s", prefix);
 }
 
diff --git a/tests/test-ovn.c b/tests/test-ovn.c
index 0ef09ab..8e9aa15 100644
--- a/tests/test-ovn.c
+++ b/tests/test-ovn.c
@@ -328,6 +328,7 @@ test_dump_symtab(struct ovs_cmdl_context *ctx OVS_UNUSED)
 ds_destroy();
 }
 
+free(nodes);
 expr_symtab_destroy();
 shash_destroy();
 }
@@ -1227,6 +1228,8 @@ test_parse_actions(struct ovs_cmdl_context *ctx 
OVS_UNUSED)
 }
 expr_destroy(prereqs2);
 
+ovnacts_free(ovnacts2.data, ovnacts2.size);
+ofpbuf_uninit();
 ds_destroy(_s);
 } else {
 printf("%s\n", error);
@@ -1234,6 +1237,7 @@ test_parse_actions(struct ovs_cmdl_context *ctx 
OVS_UNUSED)
 }
 
 expr_destroy(prereqs);
+ovnacts_free(ovnacts.data, ovnacts.size);
 ofpbuf_uninit();
 }
 ds_destroy();
-- 
2.7.4

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] ovn-controller: Fix match crieria for dynamic mac binding flows

2016-09-03 Thread Ryan Moats 


Chandra Sekhar Vejendla/San Jose/IBM@IBMUS wrote on 09/02/2016 02:34:35 PM:

> From: Chandra Sekhar Vejendla/San Jose/IBM@IBMUS
> To: dev@openvswitch.org
> Cc: Chandra Sekhar Vejendla/San Jose/IBM@IBMUS, Ryan
Moats/Omaha/IBM@IBMUS
> Date: 09/02/2016 02:35 PM
> Subject: [PATCH] ovn-controller: Fix match crieria for dynamic mac
> binding flows
>
> match struct is not initialized before adding flows for each entry in
> mac_bindings table. This results in incorrect match criteria.
>
> Signed-off-by: Chandra Sekhar Vejendla <csvej...@us.ibm.com>
> Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> Co-authored-by: Ryan Moats <rmo...@us.ibm.com>
> ---
>  ovn/controller/lflow.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/ovn/controller/lflow.c b/ovn/controller/lflow.c
> index efac5b3..fa639ae 100644
> --- a/ovn/controller/lflow.c
> +++ b/ovn/controller/lflow.c
> @@ -399,12 +399,12 @@ add_neighbor_flows(struct controller_ctx *ctx,
> struct hmap *flow_table)
>  {
>  struct ofpbuf ofpacts;
> -struct match match;
> -match_init_catchall();
>  ofpbuf_init(, 0);
>
>  const struct sbrec_mac_binding *b;
>  SBREC_MAC_BINDING_FOR_EACH (b, ctx->ovnsb_idl) {
> +struct match match;
> +match_init_catchall();
>      consider_neighbor_flow(lports, b, , , flow_table);
>  }
>
> --

This looks obvious to me, so ...

Acked-by: Ryan Moats <rmo...@us.ibm.com>

Side question: I'm not seeing a unit test case around flows and mac
bindings
that would have caught this.  Is this actually missing or am I just blind
today? :)
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] ovn: fix ovn-northd leaks in build_acl

2016-09-01 Thread Ryan Moats 
"dev"  wrote on 08/19/2016 08:06:50 PM:

> From: Ramu Ramamurthy 
> To: dev@openvswitch.org
> Cc: Suryanarayan Ramamurthy/San Jose/IBM@IBMUS
> Date: 08/19/2016 08:07 PM
> Subject: [ovs-dev] [PATCH] ovn: fix ovn-northd leaks in build_acl
> Sent by: "dev" 
>
> From: Ramu Ramamurthy 
>
> The following leaks are due to missing ds_destroy in a few
> places in build_acl.
>
> 5,850 bytes in 50 blocks are definitely lost in loss record 93 of 93
>at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-
> amd64-linux.so)
>by 0x4C2BACB: realloc (in /usr/lib64/valgrind/vgpreload_memcheck-
> amd64-linux.so)
>by 0x449507: xrealloc (util.c:123)
>by 0x42CC73: ds_reserve (dynamic-string.c:63)
>by 0x42D08F: ds_put_format_valist (dynamic-string.c:161)
>by 0x42D176: ds_put_format (dynamic-string.c:142)
>by 0x40D380: build_acls (ovn-northd.c:2320)
>by 0x40D380: build_lswitch_flows.constprop.36 (ovn-northd.c:2472)
>by 0x4072D9: build_lflows (ovn-northd.c:3845)
>by 0x4072D9: ovnnb_db_run (ovn-northd.c:3971)
>by 0x4072D9: main (ovn-northd.c:4375)
>
> 9,360 bytes in 72 blocks are definitely lost in loss record 93 of 93
>at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-
> amd64-linux.so)
>by 0x4C2BACB: realloc (in /usr/lib64/valgrind/vgpreload_memcheck-
> amd64-linux.so)
>by 0x449507: xrealloc (util.c:123)
>by 0x42CC73: ds_reserve (dynamic-string.c:63)
>by 0x42D08F: ds_put_format_valist (dynamic-string.c:161)
>by 0x42D176: ds_put_format (dynamic-string.c:142)
>by 0x40D505: build_acls (ovn-northd.c:2346)
>by 0x40D505: build_lswitch_flows.constprop.36 (ovn-northd.c:2472)
>by 0x4072D9: build_lflows (ovn-northd.c:3845)
>by 0x4072D9: ovnnb_db_run (ovn-northd.c:3971)
>by 0x4072D9: main (ovn-northd.c:4375)
> ---
>  ovn/northd/ovn-northd.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/ovn/northd/ovn-northd.c b/ovn/northd/ovn-northd.c
> index 625937d..756d188 100644
> --- a/ovn/northd/ovn-northd.c
> +++ b/ovn/northd/ovn-northd.c
> @@ -2324,6 +2324,7 @@ build_acls(struct ovn_datapath *od, struct hmap
*lflows)
>  ovn_lflow_add(
>  lflows, od, S_SWITCH_OUT_ACL, 34000, ds_cstr
(),
>  actions);
> +ds_destroy();
>  }
>  }
>
> @@ -2350,6 +2351,7 @@ build_acls(struct ovn_datapath *od, struct hmap
*lflows)
>  ovn_lflow_add(
>  lflows, od, S_SWITCH_OUT_ACL, 34000, ds_cstr
(),
>  actions);
> +ds_destroy();
>  }
>  }
>  }
> --

I had given a base ack for this previously, and I can say that as of this
message that this is the last *definitive* loss of memory in ovn-northd,
ovn-controller, and ovsdb-server.

Because of that, I'd like to see this merge into master and 2.6 before
the release is cut...

Ryan

Ryan
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] ovn-controller: Fix memory leak in recv_S_TLV_TABLE_REQUESTED().

2016-09-01 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 09/01/2016 12:02:53 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: dev@openvswitch.org
> Cc: Ben Pfaff <b...@ovn.org>, Ryan Moats/Omaha/IBM@IBMUS
> Date: 09/01/2016 12:04 PM
> Subject: [PATCH] ovn-controller: Fix memory leak in
> recv_S_TLV_TABLE_REQUESTED().
>
> Nothing freed 'reply'.  This fixes the problem.
>
> Most of this patch is moving coding around.  The essential change is that
> breaking the code that works with 'reply' out into a separate function
> makes it possible to catch all paths out of the function so that 'reply'
> can be freed in one place.
>
> Reported-by: Ryan Moats <rmo...@us.ibm.com>
> Signed-off-by: Ben Pfaff <b...@ovn.org>
> ---

That appears to do the trick as well and it is cleaner, so ... :)

Acked-by: Ryan Moats <rmo...@us.ibm.com>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH] Fix leak in patched_datapaths processing.

2016-09-01 Thread Ryan Moats 
When unpersisting patch_datapaths, missed that the key field
needed to be freed to avoid a memory leak.  This patch fixes
same.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/ovn-controller.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c
index c2e667b..c4f02bc 100644
--- a/ovn/controller/ovn-controller.c
+++ b/ovn/controller/ovn-controller.c
@@ -477,6 +477,7 @@ main(int argc, char *argv[])
 HMAP_FOR_EACH_SAFE (pd_cur_node, pd_next_node, hmap_node,
 _datapaths) {
 hmap_remove(_datapaths, _cur_node->hmap_node);
+free(pd_cur_node->key);
 free(pd_cur_node);
 }
 hmap_destroy(_datapaths);
-- 
2.7.4 (Apple Git-66)

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH] Close memory leak when processing S_TLV_TABLE_REQUESTED.

2016-09-01 Thread Ryan Moats 
Found by running valgrind on "ovn-controller -
Chassis external_ids" unit test case:
  24 bytes in 1 blocks are definitely lost in loss record 102 of 180
 at 0x4C2DBB6: malloc (vg_replace_malloc.c:299)
 by 0x4916A4: xmalloc (util.c:112)
 by 0x47278C: decode_tlv_table_mappings (ofp-util.c:10077)
 by 0x472A3A: ofputil_decode_tlv_table_reply (ofp-util.c:10159)
 by 0x40C2B1: recv_S_TLV_TABLE_REQUESTED (ofctrl.c:209)
 by 0x40C2B1: ofctrl_run (ofctrl.c:471)
 by 0x406C8F: main (ovn-controller.c:439)

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/ofctrl.c | 4 
 1 file changed, 4 insertions(+)

diff --git a/ovn/controller/ofctrl.c b/ovn/controller/ofctrl.c
index d8e111d..0493e0d 100644
--- a/ovn/controller/ofctrl.c
+++ b/ovn/controller/ofctrl.c
@@ -210,6 +210,7 @@ recv_S_TLV_TABLE_REQUESTED(const struct ofp_header *oh, 
enum ofptype type)
 if (error) {
 VLOG_ERR("failed to decode TLV table request (%s)",
  ofperr_to_string(error));
+ofputil_uninit_tlv_table();
 goto error;
 }
 
@@ -227,10 +228,12 @@ recv_S_TLV_TABLE_REQUESTED(const struct ofp_header *oh, 
enum ofptype type)
  "unsupported index %"PRIu16,
  map->option_class, map->option_type,
  map->option_len, map->index);
+ofputil_uninit_tlv_table();
 goto error;
 } else {
 mff_ovn_geneve = MFF_TUN_METADATA0 + map->index;
 state = S_CLEAR_FLOWS;
+ofputil_uninit_tlv_table();
 return;
 }
 }
@@ -239,6 +242,7 @@ recv_S_TLV_TABLE_REQUESTED(const struct ofp_header *oh, 
enum ofptype type)
 md_free &= ~(UINT64_C(1) << map->index);
 }
 }
+ofputil_uninit_tlv_table();
 
 VLOG_DBG("OVN Geneve option not found");
 if (!md_free) {
-- 
2.7.4 (Apple Git-66)

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] ovn-controller: Convert encaps module back to full processing

2016-08-31 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 08/31/2016 04:26:37 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: ovs dev <dev@openvswitch.org>, Jesse Gross <je...@kernel.org>
> Date: 08/31/2016 04:26 PM
> Subject: Re: [ovs-dev] [PATCH] ovn-controller: Convert encaps module
> back to full processing
>
> On Wed, Aug 31, 2016 at 12:59:05PM -0500, Ryan Moats wrote:
> > Ben Pfaff <b...@ovn.org> wrote on 08/31/2016 12:46:17 PM:
> >
> > > From: Ben Pfaff <b...@ovn.org>
> > > To: Jesse Gross <je...@kernel.org>
> > > Cc: Ryan Moats/Omaha/IBM@IBMUS, ovs dev <dev@openvswitch.org>
> > > Date: 08/31/2016 12:46 PM
> > > Subject: Re: [ovs-dev] [PATCH] ovn-controller: Convert encaps module
> > > back to full processing
> > >
> > > On Tue, Aug 30, 2016 at 09:04:10AM -0700, Jesse Gross wrote:
> > > > On Sun, Aug 28, 2016 at 3:51 PM, Ryan Moats <rmo...@us.ibm.com>
wrote:
> > > > > diff --git a/ovn/controller/encaps.c b/ovn/controller/encaps.c
> > > > > index d99ba05..87f 100644
> > > > > --- a/ovn/controller/encaps.c
> > > > > +++ b/ovn/controller/encaps.c
> > > > > +SBREC_CHASSIS_FOR_EACH (chassis_rec, ctx->ovnsb_idl) {
> > > > > +for (int i = 0; i < chassis_rec->n_encaps; i++) {
> > > > > +encap_rec = chassis_rec->encaps[i];
> > > > > +
> > > > > +struct encap_hash_node *encap_hash_node;
> > > > > +encap_hash_node =
> > > lookup_encap_uuid(_rec->header_.uuid);
> > > > > +if (encap_hash_node) {
> > > > > +/* A change might have invalidated our mapping.
> > > Process the
> > > > > + * new version and then iterate over everything
> > > to see if it
> > > > > + * is OK. */
> > > > > +delete_encap_uuid(encap_hash_node);
> > > > > +poll_immediate_wake();
> > > > >  }
> > > >
> > > > Doesn't this result in essentially infinite wakeups? It used to be
> > > > that this loop would run only when a chassis was
add/removed/changed
> > > > and so the if (encap_hash_node) condition would only trigger when
an
> > > > existing chassis is modified. However, after this change every
wakeup
> > > > will loop through all chassises and any existing ones will trigger
> > > > another wakeup and loop, etc.
> > > >
> > > > In general, I don't think it makes sense to remove incremental
> > > > processing without removing persistent state. The result ends up
being
> > > > not very logical and actually more complicated.
> > >
> > > I want to second Jesse's feedback here.  I think that this should
really
> > > be stateless, not half-incremental.
> >
> > Yes that poll_immediate_wake does result in infinite wakeups and yes
> > it has been removed from v2 of the patch set.
> >
> > I explained the situation in [1] and that "cry for help" still holds...
> >
> > [1] http://openvswitch.org/pipermail/dev/2016-August/078692.html
>
> I sent my suggestion as a 2-patch series:
> https://patchwork.ozlabs.org/patch/664685/
> https://patchwork.ozlabs.org/patch/664686/
>

I will give them a review first chance I get...

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [ovs-dev, v2, 3/4] Unpersist data structures for address sets.

2016-08-31 Thread Ryan Moats 


Ben Pfaff <b...@ovn.org> wrote on 08/31/2016 02:45:36 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/31/2016 02:45 PM
> Subject: Re: [ovs-dev] [ovs-dev, v2,3/4] Unpersist data structures
> for address sets.
>
> On Wed, Aug 31, 2016 at 01:51:51PM -0500, Ryan Moats wrote:
> >
> >
> > "dev" <dev-boun...@openvswitch.org> wrote on 08/31/2016 12:53:27 PM:
> >
> > > From: Ryan Moats/Omaha/IBM@IBMUS
> > > To: Ben Pfaff <b...@ovn.org>
> > > Cc: dev@openvswitch.org
> > > Date: 08/31/2016 12:53 PM
> > > Subject: Re: [ovs-dev] [ovs-dev, v2, 3/4] Unpersist data structures
> > > for address sets.
> > > Sent by: "dev" <dev-boun...@openvswitch.org>
> > >
> > >
> > > Ben Pfaff <b...@ovn.org> wrote on 08/31/2016 12:38:34 PM:
> > >
> > > > From: Ben Pfaff <b...@ovn.org>
> > > > To: Ryan Moats/Omaha/IBM@IBMUS
> > > > Cc: dev@openvswitch.org
> > > > Date: 08/31/2016 12:38 PM
> > > > Subject: Re: [ovs-dev,v2,3/4] Unpersist data structures for address
> > sets.
> > > >
> > > > On Wed, Aug 31, 2016 at 03:22:45PM +, Ryan Moats wrote:
> > > > > With the removal of incremental processing, it is no longer
> > > > > necessary to persist the data structures for storing address
> > > > > sets.  Simplify things by removing this complexity.
> > > > >
> > > > > Side effect: fixed a memory leak in expr_macros_destroy that
> > > > > is evidenced by this change.
> > > > >
> > > > > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> > > >
> > > > I think that this was still doing a lot more work than necessary.
The
> > > > "struct address_set"s were being created and destroyed but not used
for
> > > > anything in between.
> > > >
> > > > Here's my proposal.  Comments?
> > > >
> > > > --8<--cut here-->
8--
> > > >
> > > > From: Ryan Moats <rmo...@us.ibm.com>
> > > > Date: Wed, 31 Aug 2016 15:22:45 +
> > > > Subject: [PATCH] Unpersist data structures for address sets.
> > > >
> > > > With the removal of incremental processing, it is no longer
> > > > necessary to persist the data structures for storing address
> > > > sets.  Simplify things by removing this complexity.
> > > >
> > > > Side effect: fixed a memory leak in expr_macros_destroy that
> > > > is evidenced by this change.
> > > >
> > > > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> > > > Signed-off-by: Ben Pfaff <b...@ovn.org>
> > > > ---
> > > >  ovn/controller/lflow.c | 176 ++
> > > > +--
> > > >  ovn/lib/expr.c |   1 +
> > > >  2 files changed, 25 insertions(+), 152 deletions(-)
> > > >
> > > > diff --git a/ovn/controller/lflow.c b/ovn/controller/lflow.c
> > > > index a9c3137..efac5b3 100644
> > > > --- a/ovn/controller/lflow.c
> > > > +++ b/ovn/controller/lflow.c
> > > > @@ -38,159 +38,24 @@ VLOG_DEFINE_THIS_MODULE(lflow);
> > > >  /* Contains "struct expr_symbol"s for fields supported by OVN
lflows.
> > */
> > > >  static struct shash symtab;
> > > >
> > > > -/* Contains an internal expr datastructure that represents an
address
> > > set. */
> > > > -static struct shash expr_address_sets;
> > > > -
> > > >  void
> > > >  lflow_init(void)
> > > >  {
> > > >  ovn_init_symtab();
> > > > -shash_init(_address_sets);
> > > > -}
> > > > -
> > > > -/* Details of an address set currently in address_sets. We keep a
> > cached
> > > > - * copy of sets still in their string form here to make it easier
to
> > > compare
> > > > - * with the current values in the OVN_Southbound database. */
> > > > -struct address_set {
> > > > -char **addresses;
> > > > -size_t n_addresses;
> > > > -};
> > > > -
> > > > -/* struct address_set instances for address sets currently in the
> > > symtab,
> > > > - * hashed on the address set name. */
> > > > -st

Re: [ovs-dev] [ovs-dev, v2, 3/4] Unpersist data structures for address sets.

2016-08-31 Thread Ryan Moats 


"dev" <dev-boun...@openvswitch.org> wrote on 08/31/2016 12:53:27 PM:

> From: Ryan Moats/Omaha/IBM@IBMUS
> To: Ben Pfaff <b...@ovn.org>
> Cc: dev@openvswitch.org
> Date: 08/31/2016 12:53 PM
> Subject: Re: [ovs-dev] [ovs-dev, v2, 3/4] Unpersist data structures
> for address sets.
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
>
> Ben Pfaff <b...@ovn.org> wrote on 08/31/2016 12:38:34 PM:
>
> > From: Ben Pfaff <b...@ovn.org>
> > To: Ryan Moats/Omaha/IBM@IBMUS
> > Cc: dev@openvswitch.org
> > Date: 08/31/2016 12:38 PM
> > Subject: Re: [ovs-dev,v2,3/4] Unpersist data structures for address
sets.
> >
> > On Wed, Aug 31, 2016 at 03:22:45PM +, Ryan Moats wrote:
> > > With the removal of incremental processing, it is no longer
> > > necessary to persist the data structures for storing address
> > > sets.  Simplify things by removing this complexity.
> > >
> > > Side effect: fixed a memory leak in expr_macros_destroy that
> > > is evidenced by this change.
> > >
> > > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> >
> > I think that this was still doing a lot more work than necessary.  The
> > "struct address_set"s were being created and destroyed but not used for
> > anything in between.
> >
> > Here's my proposal.  Comments?
> >
> > --8<--cut here-->8--
> >
> > From: Ryan Moats <rmo...@us.ibm.com>
> > Date: Wed, 31 Aug 2016 15:22:45 +
> > Subject: [PATCH] Unpersist data structures for address sets.
> >
> > With the removal of incremental processing, it is no longer
> > necessary to persist the data structures for storing address
> > sets.  Simplify things by removing this complexity.
> >
> > Side effect: fixed a memory leak in expr_macros_destroy that
> > is evidenced by this change.
> >
> > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> > Signed-off-by: Ben Pfaff <b...@ovn.org>
> > ---
> >  ovn/controller/lflow.c | 176 ++
> > +--
> >  ovn/lib/expr.c |   1 +
> >  2 files changed, 25 insertions(+), 152 deletions(-)
> >
> > diff --git a/ovn/controller/lflow.c b/ovn/controller/lflow.c
> > index a9c3137..efac5b3 100644
> > --- a/ovn/controller/lflow.c
> > +++ b/ovn/controller/lflow.c
> > @@ -38,159 +38,24 @@ VLOG_DEFINE_THIS_MODULE(lflow);
> >  /* Contains "struct expr_symbol"s for fields supported by OVN lflows.
*/
> >  static struct shash symtab;
> >
> > -/* Contains an internal expr datastructure that represents an address
> set. */
> > -static struct shash expr_address_sets;
> > -
> >  void
> >  lflow_init(void)
> >  {
> >  ovn_init_symtab();
> > -shash_init(_address_sets);
> > -}
> > -
> > -/* Details of an address set currently in address_sets. We keep a
cached
> > - * copy of sets still in their string form here to make it easier to
> compare
> > - * with the current values in the OVN_Southbound database. */
> > -struct address_set {
> > -char **addresses;
> > -size_t n_addresses;
> > -};
> > -
> > -/* struct address_set instances for address sets currently in the
> symtab,
> > - * hashed on the address set name. */
> > -static struct shash local_address_sets =
> > SHASH_INITIALIZER(_address_sets);
> > -
> > -static int
> > -addr_cmp(const void *p1, const void *p2)
> > -{
> > -const char *s1 = p1;
> > -const char *s2 = p2;
> > -return strcmp(s1, s2);
> > -}
> > -
> > -/* Return true if the address sets match, false otherwise. */
> > -static bool
> > -address_sets_match(const struct address_set *addr_set,
> > -   const struct sbrec_address_set *addr_set_rec)
> > -{
> > -char **addrs1;
> > -char **addrs2;
> > -
> > -if (addr_set->n_addresses != addr_set_rec->n_addresses) {
> > -return false;
> > -}
> > -size_t n_addresses = addr_set->n_addresses;
> > -
> > -addrs1 = xmemdup(addr_set->addresses,
> > - n_addresses * sizeof addr_set->addresses[0]);
> > -addrs2 = xmemdup(addr_set_rec->addresses,
> > - n_addresses * sizeof addr_set_rec->addresses[0]);
> > -
> > -qsort(addrs1, n_addresses, sizeof *addrs1, addr_cmp);
> > -qsort(addrs2, n_addresses, sizeof *addrs2, addr_cmp);
> > -

Re: [ovs-dev] [PATCH] ovn-controller: Convert encaps module back to full processing

2016-08-31 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 08/31/2016 12:46:17 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Jesse Gross <je...@kernel.org>
> Cc: Ryan Moats/Omaha/IBM@IBMUS, ovs dev <dev@openvswitch.org>
> Date: 08/31/2016 12:46 PM
> Subject: Re: [ovs-dev] [PATCH] ovn-controller: Convert encaps module
> back to full processing
>
> On Tue, Aug 30, 2016 at 09:04:10AM -0700, Jesse Gross wrote:
> > On Sun, Aug 28, 2016 at 3:51 PM, Ryan Moats <rmo...@us.ibm.com> wrote:
> > > diff --git a/ovn/controller/encaps.c b/ovn/controller/encaps.c
> > > index d99ba05..87f 100644
> > > --- a/ovn/controller/encaps.c
> > > +++ b/ovn/controller/encaps.c
> > > +SBREC_CHASSIS_FOR_EACH (chassis_rec, ctx->ovnsb_idl) {
> > > +for (int i = 0; i < chassis_rec->n_encaps; i++) {
> > > +encap_rec = chassis_rec->encaps[i];
> > > +
> > > +struct encap_hash_node *encap_hash_node;
> > > +encap_hash_node =
> lookup_encap_uuid(_rec->header_.uuid);
> > > +if (encap_hash_node) {
> > > +/* A change might have invalidated our mapping.
> Process the
> > > + * new version and then iterate over everything
> to see if it
> > > + * is OK. */
> > > +delete_encap_uuid(encap_hash_node);
> > > +poll_immediate_wake();
> > >  }
> >
> > Doesn't this result in essentially infinite wakeups? It used to be
> > that this loop would run only when a chassis was add/removed/changed
> > and so the if (encap_hash_node) condition would only trigger when an
> > existing chassis is modified. However, after this change every wakeup
> > will loop through all chassises and any existing ones will trigger
> > another wakeup and loop, etc.
> >
> > In general, I don't think it makes sense to remove incremental
> > processing without removing persistent state. The result ends up being
> > not very logical and actually more complicated.
>
> I want to second Jesse's feedback here.  I think that this should really
> be stateless, not half-incremental.

Yes that poll_immediate_wake does result in infinite wakeups and yes
it has been removed from v2 of the patch set.

I explained the situation in [1] and that "cry for help" still holds...

[1] http://openvswitch.org/pipermail/dev/2016-August/078692.html
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [ovs-dev, v2, 3/4] Unpersist data structures for address sets.

2016-08-31 Thread Ryan Moats 

Ben Pfaff <b...@ovn.org> wrote on 08/31/2016 12:38:34 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/31/2016 12:38 PM
> Subject: Re: [ovs-dev,v2,3/4] Unpersist data structures for address sets.
>
> On Wed, Aug 31, 2016 at 03:22:45PM +, Ryan Moats wrote:
> > With the removal of incremental processing, it is no longer
> > necessary to persist the data structures for storing address
> > sets.  Simplify things by removing this complexity.
> >
> > Side effect: fixed a memory leak in expr_macros_destroy that
> > is evidenced by this change.
> >
> > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
>
> I think that this was still doing a lot more work than necessary.  The
> "struct address_set"s were being created and destroyed but not used for
> anything in between.
>
> Here's my proposal.  Comments?
>
> --8<--cut here-->8--
>
> From: Ryan Moats <rmo...@us.ibm.com>
> Date: Wed, 31 Aug 2016 15:22:45 +
> Subject: [PATCH] Unpersist data structures for address sets.
>
> With the removal of incremental processing, it is no longer
> necessary to persist the data structures for storing address
> sets.  Simplify things by removing this complexity.
>
> Side effect: fixed a memory leak in expr_macros_destroy that
> is evidenced by this change.
>
> Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> Signed-off-by: Ben Pfaff <b...@ovn.org>
> ---
>  ovn/controller/lflow.c | 176 ++
> +--
>  ovn/lib/expr.c |   1 +
>  2 files changed, 25 insertions(+), 152 deletions(-)
>
> diff --git a/ovn/controller/lflow.c b/ovn/controller/lflow.c
> index a9c3137..efac5b3 100644
> --- a/ovn/controller/lflow.c
> +++ b/ovn/controller/lflow.c
> @@ -38,159 +38,24 @@ VLOG_DEFINE_THIS_MODULE(lflow);
>  /* Contains "struct expr_symbol"s for fields supported by OVN lflows. */
>  static struct shash symtab;
>
> -/* Contains an internal expr datastructure that represents an address
set. */
> -static struct shash expr_address_sets;
> -
>  void
>  lflow_init(void)
>  {
>  ovn_init_symtab();
> -shash_init(_address_sets);
> -}
> -
> -/* Details of an address set currently in address_sets. We keep a cached
> - * copy of sets still in their string form here to make it easier to
compare
> - * with the current values in the OVN_Southbound database. */
> -struct address_set {
> -char **addresses;
> -size_t n_addresses;
> -};
> -
> -/* struct address_set instances for address sets currently in the
symtab,
> - * hashed on the address set name. */
> -static struct shash local_address_sets =
> SHASH_INITIALIZER(_address_sets);
> -
> -static int
> -addr_cmp(const void *p1, const void *p2)
> -{
> -const char *s1 = p1;
> -const char *s2 = p2;
> -return strcmp(s1, s2);
> -}
> -
> -/* Return true if the address sets match, false otherwise. */
> -static bool
> -address_sets_match(const struct address_set *addr_set,
> -   const struct sbrec_address_set *addr_set_rec)
> -{
> -char **addrs1;
> -char **addrs2;
> -
> -if (addr_set->n_addresses != addr_set_rec->n_addresses) {
> -return false;
> -}
> -size_t n_addresses = addr_set->n_addresses;
> -
> -addrs1 = xmemdup(addr_set->addresses,
> - n_addresses * sizeof addr_set->addresses[0]);
> -addrs2 = xmemdup(addr_set_rec->addresses,
> - n_addresses * sizeof addr_set_rec->addresses[0]);
> -
> -qsort(addrs1, n_addresses, sizeof *addrs1, addr_cmp);
> -qsort(addrs2, n_addresses, sizeof *addrs2, addr_cmp);
> -
> -bool res = true;
> -size_t i;
> -for (i = 0; i <  n_addresses; i++) {
> -if (strcmp(addrs1[i], addrs2[i])) {
> -res = false;
> -break;
> -}
> -}
> -
> -free(addrs1);
> -free(addrs2);
> -
> -return res;
>  }
>
> +/* Iterate address sets in the southbound database.  Create and update
the
> + * corresponding symtab entries as necessary. */
>  static void
> -address_set_destroy(struct address_set *addr_set)
> -{
> -size_t i;
> -for (i = 0; i < addr_set->n_addresses; i++) {
> -free(addr_set->addresses[i]);
> -}
> -if (addr_set->n_addresses) {
> -free(addr_set->addresses);
> -}
> -free(addr_set);
> -}
> +update_address_sets(struct controller_ctx *ctx,
> +struct shash *expr_address_sets_

[ovs-dev] [PATCH v2 4/4] ovn-controller: Convert encaps module back to full processing

2016-08-31 Thread Ryan Moats 
This patch converts the encaps module back to full processing, but
does not remove all persistence of associated data strcutures.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/encaps.c | 33 ++---
 1 file changed, 14 insertions(+), 19 deletions(-)

diff --git a/ovn/controller/encaps.c b/ovn/controller/encaps.c
index d99ba05..384e8e7 100644
--- a/ovn/controller/encaps.c
+++ b/ovn/controller/encaps.c
@@ -412,25 +412,20 @@ encaps_run(struct controller_ctx *ctx, const struct 
ovsrec_bridge *br_int,
 /* Maintain a mapping backwards from encap entries to their parent
  * chassis. Most changes happen at the encap row entry but tunnels need
  * to be established on the basis of the overall chassis. */
-SBREC_CHASSIS_FOR_EACH_TRACKED (chassis_rec, ctx->ovnsb_idl) {
-/* Defer deletion of mapping until we have cleaned up associated
- * ports. */
-if (!sbrec_chassis_is_deleted(chassis_rec)) {
-for (int i = 0; i < chassis_rec->n_encaps; i++) {
-encap_rec = chassis_rec->encaps[i];
-
-struct encap_hash_node *encap_hash_node;
-encap_hash_node = lookup_encap_uuid(_rec->header_.uuid);
-if (encap_hash_node) {
-/* A change might have invalidated our mapping. Process the
- * new version and then iterate over everything to see if 
it
- * is OK. */
-delete_encap_uuid(encap_hash_node);
-poll_immediate_wake();
-}
-
-insert_encap_uuid(_rec->header_.uuid, chassis_rec);
+SBREC_CHASSIS_FOR_EACH (chassis_rec, ctx->ovnsb_idl) {
+for (int i = 0; i < chassis_rec->n_encaps; i++) {
+encap_rec = chassis_rec->encaps[i];
+
+struct encap_hash_node *encap_hash_node;
+encap_hash_node = lookup_encap_uuid(_rec->header_.uuid);
+if (encap_hash_node) {
+/* A change might have invalidated our mapping. Process the
+ * new version and then iterate over everything to see if it
+ * is OK. */
+delete_encap_uuid(encap_hash_node);
 }
+
+insert_encap_uuid(_rec->header_.uuid, chassis_rec);
 }
 }
 
@@ -440,7 +435,7 @@ encaps_run(struct controller_ctx *ctx, const struct 
ovsrec_bridge *br_int,
  * might actually result in the creation of a different type tunnel if
  * that type is preferred. That's OK - when we process the other encap
  * rows, we'll just skip over the new tunnels. */
-SBREC_ENCAP_FOR_EACH_TRACKED (encap_rec, ctx->ovnsb_idl) {
+SBREC_ENCAP_FOR_EACH (encap_rec, ctx->ovnsb_idl) {
 struct encap_hash_node *encap_hash_node;
 struct chassis_hash_node *chassis_hash_node;
 const struct ovsrec_port *port_rec = NULL;
-- 
2.7.4

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v2 3/4] Unpersist data structures for address sets.

2016-08-31 Thread Ryan Moats 
With the removal of incremental processing, it is no longer
necessary to persist the data structures for storing address
sets.  Simplify things by removing this complexity.

Side effect: fixed a memory leak in expr_macros_destroy that
is evidenced by this change.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/lflow.c | 166 -
 ovn/lib/expr.c |   1 +
 2 files changed, 42 insertions(+), 125 deletions(-)

diff --git a/ovn/controller/lflow.c b/ovn/controller/lflow.c
index dc69047..5713c46 100644
--- a/ovn/controller/lflow.c
+++ b/ovn/controller/lflow.c
@@ -38,14 +38,10 @@ VLOG_DEFINE_THIS_MODULE(lflow);
 /* Contains "struct expr_symbol"s for fields supported by OVN lflows. */
 static struct shash symtab;
 
-/* Contains an internal expr datastructure that represents an address set. */
-static struct shash expr_address_sets;
-
 void
 lflow_init(void)
 {
 ovn_init_symtab();
-shash_init(_address_sets);
 }
 
 /* Details of an address set currently in address_sets. We keep a cached
@@ -56,54 +52,6 @@ struct address_set {
 size_t n_addresses;
 };
 
-/* struct address_set instances for address sets currently in the symtab,
- * hashed on the address set name. */
-static struct shash local_address_sets = 
SHASH_INITIALIZER(_address_sets);
-
-static int
-addr_cmp(const void *p1, const void *p2)
-{
-const char *s1 = p1;
-const char *s2 = p2;
-return strcmp(s1, s2);
-}
-
-/* Return true if the address sets match, false otherwise. */
-static bool
-address_sets_match(const struct address_set *addr_set,
-   const struct sbrec_address_set *addr_set_rec)
-{
-char **addrs1;
-char **addrs2;
-
-if (addr_set->n_addresses != addr_set_rec->n_addresses) {
-return false;
-}
-size_t n_addresses = addr_set->n_addresses;
-
-addrs1 = xmemdup(addr_set->addresses,
- n_addresses * sizeof addr_set->addresses[0]);
-addrs2 = xmemdup(addr_set_rec->addresses,
- n_addresses * sizeof addr_set_rec->addresses[0]);
-
-qsort(addrs1, n_addresses, sizeof *addrs1, addr_cmp);
-qsort(addrs2, n_addresses, sizeof *addrs2, addr_cmp);
-
-bool res = true;
-size_t i;
-for (i = 0; i <  n_addresses; i++) {
-if (strcmp(addrs1[i], addrs2[i])) {
-res = false;
-break;
-}
-}
-
-free(addrs1);
-free(addrs2);
-
-return res;
-}
-
 static void
 address_set_destroy(struct address_set *addr_set)
 {
@@ -118,79 +66,34 @@ address_set_destroy(struct address_set *addr_set)
 }
 
 static void
-update_address_sets(struct controller_ctx *ctx)
-{
-/* Remember the names of all address sets currently in expr_address_sets
- * so we can detect address sets that have been deleted. */
-struct sset cur_addr_set_names = SSET_INITIALIZER(_addr_set_names);
-
-struct shash_node *node;
-SHASH_FOR_EACH (node, _address_sets) {
-sset_add(_addr_set_names, node->name);
-}
+update_address_sets(struct controller_ctx *ctx,
+struct shash *local_address_sets_p,
+struct shash *expr_address_sets_p)
 
+{
 /* Iterate address sets in the southbound database.  Create and update the
  * corresponding symtab entries as necessary. */
 const struct sbrec_address_set *addr_set_rec;
 SBREC_ADDRESS_SET_FOR_EACH (addr_set_rec, ctx->ovnsb_idl) {
-struct address_set *addr_set =
-shash_find_data(_address_sets, addr_set_rec->name);
-
-bool create_set = false;
-if (addr_set) {
-/* This address set has already been added.  We must determine
- * if the symtab entry needs to be updated due to a change. */
-sset_find_and_delete(_addr_set_names, addr_set_rec->name);
-if (!address_sets_match(addr_set, addr_set_rec)) {
-shash_find_and_delete(_address_sets, addr_set_rec->name);
-expr_macros_remove(_address_sets, addr_set_rec->name);
-address_set_destroy(addr_set);
-addr_set = NULL;
-create_set = true;
+/* Create a symbol that resolves to the full set of addresses.
+ * Store it in address_sets to remember that we created this
+ * symbol. */
+struct address_set *addr_set = xzalloc(sizeof *addr_set);
+addr_set->n_addresses = addr_set_rec->n_addresses;
+if (addr_set_rec->n_addresses) {
+addr_set->addresses = xmalloc(addr_set_rec->n_addresses
+  * sizeof addr_set->addresses[0]);
+size_t i;
+for (i = 0; i < addr_set_rec->n_addresses; i++) {
+addr_set->addresses[i] = xstrdup(addr_set_rec->addresses[i]);
 }
-} else {
-/* This address set is not yet in the symtab, so add i

[ovs-dev] [PATCH v2 2/4] ovn-controller: add quiet mode

2016-08-31 Thread Ryan Moats 
As discussed in [1], what the incremental processing code
actually accomplished was that the ovn-controller would
be "quiet" and not burn CPU when things weren't changing.
This patch set recreates this state by calculating whether
changes have occured that would require a full calculation
to be performed.  It does this by persisting a copy of
the localvif_to_ofport and tunnel information in the
controller module, rather than in the physical.c module
as was the case with previous commits.

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/ofctrl.c |   2 +
 ovn/controller/ovn-controller.c |  73 -
 ovn/controller/ovn-controller.h |   1 +
 ovn/controller/patch.c  |  11 +
 ovn/controller/physical.c   | 100 
 ovn/controller/physical.h   |   5 ++
 6 files changed, 131 insertions(+), 61 deletions(-)

diff --git a/ovn/controller/ofctrl.c b/ovn/controller/ofctrl.c
index d8e111d..5b1434e 100644
--- a/ovn/controller/ofctrl.c
+++ b/ovn/controller/ofctrl.c
@@ -366,6 +366,8 @@ run_S_CLEAR_FLOWS(void)
 queue_msg(encode_group_mod());
 ofputil_uninit_group_mod();
 
+force_full_process();
+
 /* Clear existing groups, to match the state of the switch. */
 if (groups) {
 ovn_group_table_clear(groups, true);
diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c
index c2e667b..cb0ae40 100644
--- a/ovn/controller/ovn-controller.c
+++ b/ovn/controller/ovn-controller.c
@@ -304,6 +304,23 @@ get_nb_cfg(struct ovsdb_idl *idl)
 return sb ? sb->nb_cfg : 0;
 }
 
+/* Contains mapping of localvifs to OF ports for detecting if the physical
+ * configuration of the switch has changed. */
+static struct simap localvif_to_ofport =
+SIMAP_INITIALIZER(_to_ofport);
+
+/* Contains the list of known tunnels. */
+static struct hmap tunnels = HMAP_INITIALIZER();
+
+/* The last sequence number seen from the southbound IDL. */
+static unsigned int seqno = 0;
+
+void
+force_full_process(void)
+{
+seqno = 0;
+}
+
 int
 main(int argc, char *argv[])
 {
@@ -427,39 +444,51 @@ main(int argc, char *argv[])
 _lports);
 }
 
+enum mf_field_id mff_ovn_geneve;
+bool physical_change = true;
 if (br_int && chassis_id) {
+mff_ovn_geneve = ofctrl_run(br_int);
+physical_change = detect_and_save_physical_changes(
+_to_ofport, , mff_ovn_geneve, br_int,
+chassis_id);
+unsigned int cur_seqno = ovsdb_idl_get_seqno(ovnsb_idl_loop.idl);
+
 patch_run(, br_int, chassis_id, _datapaths,
   _datapaths);
 
 static struct lport_index lports;
-static struct mcgroup_index mcgroups;
 lport_index_init(, ctx.ovnsb_idl);
-mcgroup_index_init(, ctx.ovnsb_idl);
-
-enum mf_field_id mff_ovn_geneve = ofctrl_run(br_int);
-
-pinctrl_run(, , br_int, chassis_id, _datapaths);
+pinctrl_run(, , br_int, chassis_id,
+_datapaths);
 update_ct_zones(_lports, _datapaths, _zones,
 ct_zone_bitmap);
 
-struct hmap flow_table = HMAP_INITIALIZER(_table);
-lflow_run(, , , _datapaths,
-  _datapaths, _table, _zones,
-  _table);
-
-physical_run(, mff_ovn_geneve,
- br_int, chassis_id, _zones, _table,
- _datapaths, _datapaths);
-
-ofctrl_put(_table, get_nb_cfg(ctx.ovnsb_idl));
-hmap_destroy(_table);
-if (ctx.ovnsb_idl_txn) {
-int64_t cur_cfg = ofctrl_get_cur_cfg();
-if (cur_cfg && cur_cfg != chassis->nb_cfg) {
-sbrec_chassis_set_nb_cfg(chassis, cur_cfg);
+
+if (physical_change || seqno < cur_seqno) {
+seqno = cur_seqno;
+
+static struct mcgroup_index mcgroups;
+mcgroup_index_init(, ctx.ovnsb_idl);
+
+struct hmap flow_table = HMAP_INITIALIZER(_table);
+lflow_run(, , , _datapaths,
+  _datapaths, _table, _zones,
+  _table);
+
+physical_run(, mff_ovn_geneve,
+ br_int, chassis_id, _zones, _table,
+ _datapaths, _datapaths);
+
+ofctrl_put(_table, get_nb_cfg(ctx.ovnsb_idl));
+hmap_destroy(_table);
+if (ctx.ovnsb_idl_txn) {
+int64_t cur_cfg = ofctrl_get_cur_cfg();
+if (cur_cfg && cur_cfg != chassis->nb_cfg) {
+sbrec_chassis_set_nb_cfg(chassis, cur_cfg);
+}

[ovs-dev] [PATCH v2 1/4] ovn-controller: Back out incremental processing

2016-08-31 Thread Ryan Moats 
As [1] indicates, incremental processing hasn't resulted
in an improvement worth the complexity it has added.
This patch backs out all of the code specific to incremental
processing, along with the persisting of OF flows,
logical ports, multicast groups, all_lports, local and patched
datapaths.

Persisted objects in the ovn/controller/physical.c module will
be used by a future patch set to determine if physical changes
have occurred.

Future patch sets in the series will convert
the ovn/controller/encaps.c module back to full processing
and remove the persistance of address sets in the
ovn/controller/lflow.c module.

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 include/ovn/actions.h   |   4 -
 ovn/controller/binding.c| 154 ++
 ovn/controller/binding.h|   1 -
 ovn/controller/encaps.c | 111 ++---
 ovn/controller/lflow.c  | 102 +++-
 ovn/controller/lflow.h  |   4 +-
 ovn/controller/lport.c  | 220 -
 ovn/controller/lport.h  |  24 +--
 ovn/controller/ofctrl.c | 353 +++-
 ovn/controller/ofctrl.h |  16 +-
 ovn/controller/ovn-controller.c |  61 ---
 ovn/controller/patch.c  |   6 -
 ovn/controller/physical.c   | 175 +---
 ovn/controller/physical.h   |   3 +-
 ovn/lib/actions.c   |   1 -
 tests/ovn.at|  55 +++
 tests/system-ovn.at |  45 ++---
 17 files changed, 387 insertions(+), 948 deletions(-)

diff --git a/include/ovn/actions.h b/include/ovn/actions.h
index e2a716a..b678d33 100644
--- a/include/ovn/actions.h
+++ b/include/ovn/actions.h
@@ -298,7 +298,6 @@ struct group_table {
 struct group_info {
 struct hmap_node hmap_node;
 struct ds group;
-struct uuid lflow_uuid;
 uint32_t group_id;
 };
 
@@ -412,9 +411,6 @@ struct ovnact_encode_params {
 /* A struct to figure out the group_id for group actions. */
 struct group_table *group_table;
 
-/* The logical flow uuid that drove this action. */
-struct uuid lflow_uuid;
-
 /* OVN maps each logical flow table (ltable), one-to-one, onto a physical
  * OpenFlow flow table (ptable).  A number of parameters describe this
  * mapping and data related to flow tables:
diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
index c26007d..0353a7b 100644
--- a/ovn/controller/binding.c
+++ b/ovn/controller/binding.c
@@ -30,18 +30,6 @@
 
 VLOG_DEFINE_THIS_MODULE(binding);
 
-/* A set of the iface-id values of local interfaces on this chassis. */
-static struct sset local_ids = SSET_INITIALIZER(_ids);
-
-/* When this gets set to true, the next run will re-check all binding records. 
*/
-static bool process_full_binding = false;
-
-void
-binding_reset_processing(void)
-{
-process_full_binding = true;
-}
-
 void
 binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
 {
@@ -64,16 +52,12 @@ binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
  _interface_col_ingress_policing_burst);
 }
 
-static bool
+static void
 get_local_iface_ids(const struct ovsrec_bridge *br_int,
 struct shash *lport_to_iface,
 struct sset *all_lports)
 {
 int i;
-bool changed = false;
-
-struct sset old_local_ids = SSET_INITIALIZER(_local_ids);
-sset_clone(_local_ids, _ids);
 
 for (i = 0; i < br_int->n_ports; i++) {
 const struct ovsrec_port *port_rec = br_int->ports[i];
@@ -93,53 +77,9 @@ get_local_iface_ids(const struct ovsrec_bridge *br_int,
 continue;
 }
 shash_add(lport_to_iface, iface_id, iface_rec);
-if (!sset_find_and_delete(_local_ids, iface_id)) {
-sset_add(_ids, iface_id);
-sset_add(all_lports, iface_id);
-changed = true;
-}
+sset_add(all_lports, iface_id);
 }
 }
-
-/* Any item left in old_local_ids is an ID for an interface
- * that has been removed. */
-if (!changed && !sset_is_empty(_local_ids)) {
-changed = true;
-
-const char *cur_id;
-SSET_FOR_EACH(cur_id, _local_ids) {
-sset_find_and_delete(_ids, cur_id);
-sset_find_and_delete(all_lports, cur_id);
-}
-}
-
-sset_destroy(_local_ids);
-
-return changed;
-}
-
-static struct local_datapath *
-local_datapath_lookup_by_uuid(struct hmap *hmap_p, const struct uuid *uuid)
-{
-struct local_datapath *ld;
-HMAP_FOR_EACH_WITH_HASH(ld, uuid_hmap_node, uuid_hash(uuid), hmap_p) {
-if (uuid_equals(>uuid, uuid)) {
-return ld;
-}
-}
-return NULL;
-}
-
-static void
-remove_local_datapath(struct hmap *local_datapaths, struct local_datapath *ld)
-{
-if (ld->logical_port) {
-free(ld->logical_port);

[ovs-dev] [PATCH v2 0/4] ovn-controller: Replace incremental processing with quiet mode

2016-08-31 Thread Ryan Moats 
This patch set removes incremental processing and replaces it
with quiet mode: ovn-controller first tests to see if recalculation
is necessary and if so, does a full calculation.

Side effect of this is that almost all data persistence is also reverted.
The exception is the persistence in the encaps.c module - it's not clear
how to revert that while still maintaining 100% unit test pass.

v1->v2:
  removed poll_immediate_wake() from encaps.c that will lead to
  ovn-controller pegging a cpu.

Ryan Moats (4):
  ovn-controller: Back out incremental processing
  ovn-controller: add quiet mode
  Unpersist data structures for address sets.
  ovn-controller: Convert encaps module back to full processing

 include/ovn/actions.h   |   4 -
 ovn/controller/binding.c| 154 ++
 ovn/controller/binding.h|   1 -
 ovn/controller/encaps.c | 144 +++-
 ovn/controller/lflow.c  | 258 +++--
 ovn/controller/lflow.h  |   4 +-
 ovn/controller/lport.c  | 220 -
 ovn/controller/lport.h  |  24 +--
 ovn/controller/ofctrl.c | 353 +++-
 ovn/controller/ofctrl.h |  16 +-
 ovn/controller/ovn-controller.c | 106 
 ovn/controller/ovn-controller.h |   1 +
 ovn/controller/patch.c  |  17 +-
 ovn/controller/physical.c   | 263 +++---
 ovn/controller/physical.h   |   8 +-
 ovn/lib/actions.c   |   1 -
 ovn/lib/expr.c  |   1 +
 tests/ovn.at|  55 +++
 tests/system-ovn.at |  45 ++---
 19 files changed, 548 insertions(+), 1127 deletions(-)

-- 
2.7.4

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH 3/4] Unpersist data structures for address sets.

2016-08-30 Thread Ryan Moats 
With the removal of incremental processing, it is no longer
necessary to persist the data structures for storing address
sets.  Simplify things by removing this complexity.

Side effect: fixed a memory leak in expr_macros_destroy that
is evidenced by this change.

This commit depends on f5d916cb ("ovn-controller:
Back out incremental processing").

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/lflow.c | 166 -
 ovn/lib/expr.c |   1 +
 2 files changed, 42 insertions(+), 125 deletions(-)

diff --git a/ovn/controller/lflow.c b/ovn/controller/lflow.c
index dc69047..5713c46 100644
--- a/ovn/controller/lflow.c
+++ b/ovn/controller/lflow.c
@@ -38,14 +38,10 @@ VLOG_DEFINE_THIS_MODULE(lflow);
 /* Contains "struct expr_symbol"s for fields supported by OVN lflows. */
 static struct shash symtab;
 
-/* Contains an internal expr datastructure that represents an address set. */
-static struct shash expr_address_sets;
-
 void
 lflow_init(void)
 {
 ovn_init_symtab();
-shash_init(_address_sets);
 }
 
 /* Details of an address set currently in address_sets. We keep a cached
@@ -56,54 +52,6 @@ struct address_set {
 size_t n_addresses;
 };
 
-/* struct address_set instances for address sets currently in the symtab,
- * hashed on the address set name. */
-static struct shash local_address_sets = 
SHASH_INITIALIZER(_address_sets);
-
-static int
-addr_cmp(const void *p1, const void *p2)
-{
-const char *s1 = p1;
-const char *s2 = p2;
-return strcmp(s1, s2);
-}
-
-/* Return true if the address sets match, false otherwise. */
-static bool
-address_sets_match(const struct address_set *addr_set,
-   const struct sbrec_address_set *addr_set_rec)
-{
-char **addrs1;
-char **addrs2;
-
-if (addr_set->n_addresses != addr_set_rec->n_addresses) {
-return false;
-}
-size_t n_addresses = addr_set->n_addresses;
-
-addrs1 = xmemdup(addr_set->addresses,
- n_addresses * sizeof addr_set->addresses[0]);
-addrs2 = xmemdup(addr_set_rec->addresses,
- n_addresses * sizeof addr_set_rec->addresses[0]);
-
-qsort(addrs1, n_addresses, sizeof *addrs1, addr_cmp);
-qsort(addrs2, n_addresses, sizeof *addrs2, addr_cmp);
-
-bool res = true;
-size_t i;
-for (i = 0; i <  n_addresses; i++) {
-if (strcmp(addrs1[i], addrs2[i])) {
-res = false;
-break;
-}
-}
-
-free(addrs1);
-free(addrs2);
-
-return res;
-}
-
 static void
 address_set_destroy(struct address_set *addr_set)
 {
@@ -118,79 +66,34 @@ address_set_destroy(struct address_set *addr_set)
 }
 
 static void
-update_address_sets(struct controller_ctx *ctx)
-{
-/* Remember the names of all address sets currently in expr_address_sets
- * so we can detect address sets that have been deleted. */
-struct sset cur_addr_set_names = SSET_INITIALIZER(_addr_set_names);
-
-struct shash_node *node;
-SHASH_FOR_EACH (node, _address_sets) {
-sset_add(_addr_set_names, node->name);
-}
+update_address_sets(struct controller_ctx *ctx,
+struct shash *local_address_sets_p,
+struct shash *expr_address_sets_p)
 
+{
 /* Iterate address sets in the southbound database.  Create and update the
  * corresponding symtab entries as necessary. */
 const struct sbrec_address_set *addr_set_rec;
 SBREC_ADDRESS_SET_FOR_EACH (addr_set_rec, ctx->ovnsb_idl) {
-struct address_set *addr_set =
-shash_find_data(_address_sets, addr_set_rec->name);
-
-bool create_set = false;
-if (addr_set) {
-/* This address set has already been added.  We must determine
- * if the symtab entry needs to be updated due to a change. */
-sset_find_and_delete(_addr_set_names, addr_set_rec->name);
-if (!address_sets_match(addr_set, addr_set_rec)) {
-shash_find_and_delete(_address_sets, addr_set_rec->name);
-expr_macros_remove(_address_sets, addr_set_rec->name);
-address_set_destroy(addr_set);
-addr_set = NULL;
-create_set = true;
+/* Create a symbol that resolves to the full set of addresses.
+ * Store it in address_sets to remember that we created this
+ * symbol. */
+struct address_set *addr_set = xzalloc(sizeof *addr_set);
+addr_set->n_addresses = addr_set_rec->n_addresses;
+if (addr_set_rec->n_addresses) {
+addr_set->addresses = xmalloc(addr_set_rec->n_addresses
+  * sizeof addr_set->addresses[0]);
+size_t i;
+for (i = 0; i < addr_set_rec->n_addresses; i++) {
+addr_set->addresses[i] = xstrdup(addr_set_rec->addresses[i]);

[ovs-dev] [PATCH 4/4] ovn-controller: Convert encaps module back to full processing

2016-08-30 Thread Ryan Moats 
Commit f5d916cb ("ovn-controller: Back out incremental processing")
left the conversion of the encaps module to a future patch set.
This patch converts this module back to full processing, but
does not remove all persistence of associated data strcutures.

This commit depends on f5d916cb ("ovn-controller:
Back out incremental processing").

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/encaps.c | 34 +++---
 1 file changed, 15 insertions(+), 19 deletions(-)

diff --git a/ovn/controller/encaps.c b/ovn/controller/encaps.c
index d99ba05..87f 100644
--- a/ovn/controller/encaps.c
+++ b/ovn/controller/encaps.c
@@ -412,25 +412,21 @@ encaps_run(struct controller_ctx *ctx, const struct 
ovsrec_bridge *br_int,
 /* Maintain a mapping backwards from encap entries to their parent
  * chassis. Most changes happen at the encap row entry but tunnels need
  * to be established on the basis of the overall chassis. */
-SBREC_CHASSIS_FOR_EACH_TRACKED (chassis_rec, ctx->ovnsb_idl) {
-/* Defer deletion of mapping until we have cleaned up associated
- * ports. */
-if (!sbrec_chassis_is_deleted(chassis_rec)) {
-for (int i = 0; i < chassis_rec->n_encaps; i++) {
-encap_rec = chassis_rec->encaps[i];
-
-struct encap_hash_node *encap_hash_node;
-encap_hash_node = lookup_encap_uuid(_rec->header_.uuid);
-if (encap_hash_node) {
-/* A change might have invalidated our mapping. Process the
- * new version and then iterate over everything to see if 
it
- * is OK. */
-delete_encap_uuid(encap_hash_node);
-poll_immediate_wake();
-}
-
-insert_encap_uuid(_rec->header_.uuid, chassis_rec);
+SBREC_CHASSIS_FOR_EACH (chassis_rec, ctx->ovnsb_idl) {
+for (int i = 0; i < chassis_rec->n_encaps; i++) {
+encap_rec = chassis_rec->encaps[i];
+
+struct encap_hash_node *encap_hash_node;
+encap_hash_node = lookup_encap_uuid(_rec->header_.uuid);
+if (encap_hash_node) {
+/* A change might have invalidated our mapping. Process the
+ * new version and then iterate over everything to see if it
+ * is OK. */
+delete_encap_uuid(encap_hash_node);
+poll_immediate_wake();
 }
+
+insert_encap_uuid(_rec->header_.uuid, chassis_rec);
 }
 }
 
@@ -440,7 +436,7 @@ encaps_run(struct controller_ctx *ctx, const struct 
ovsrec_bridge *br_int,
  * might actually result in the creation of a different type tunnel if
  * that type is preferred. That's OK - when we process the other encap
  * rows, we'll just skip over the new tunnels. */
-SBREC_ENCAP_FOR_EACH_TRACKED (encap_rec, ctx->ovnsb_idl) {
+SBREC_ENCAP_FOR_EACH (encap_rec, ctx->ovnsb_idl) {
 struct encap_hash_node *encap_hash_node;
 struct chassis_hash_node *chassis_hash_node;
 const struct ovsrec_port *port_rec = NULL;
-- 
2.7.4

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH 2/4] ovn-controller: add quiet mode

2016-08-30 Thread Ryan Moats 
As discussed in [1], what the incremental processing code
actually accomplished was that the ovn-controller would
be "quiet" and not burn CPU when things weren't changing.
This patch set recreates this state by calculating whether
changes have occured that would require a full calculation
to be performed.  It does this by persisting a copy of
the localvif_to_ofport and tunnel information in the
controller module, rather than in the physical.c module
as was the case with previous commits.

This commit depends on f5d916cb ("ovn-controller:
Back out incremental processing").

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/ofctrl.c |   2 +
 ovn/controller/ovn-controller.c |  73 -
 ovn/controller/ovn-controller.h |   1 +
 ovn/controller/patch.c  |  11 +
 ovn/controller/physical.c   | 100 
 ovn/controller/physical.h   |   5 ++
 6 files changed, 131 insertions(+), 61 deletions(-)

diff --git a/ovn/controller/ofctrl.c b/ovn/controller/ofctrl.c
index d8e111d..5b1434e 100644
--- a/ovn/controller/ofctrl.c
+++ b/ovn/controller/ofctrl.c
@@ -366,6 +366,8 @@ run_S_CLEAR_FLOWS(void)
 queue_msg(encode_group_mod());
 ofputil_uninit_group_mod();
 
+force_full_process();
+
 /* Clear existing groups, to match the state of the switch. */
 if (groups) {
 ovn_group_table_clear(groups, true);
diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c
index c2e667b..cb0ae40 100644
--- a/ovn/controller/ovn-controller.c
+++ b/ovn/controller/ovn-controller.c
@@ -304,6 +304,23 @@ get_nb_cfg(struct ovsdb_idl *idl)
 return sb ? sb->nb_cfg : 0;
 }
 
+/* Contains mapping of localvifs to OF ports for detecting if the physical
+ * configuration of the switch has changed. */
+static struct simap localvif_to_ofport =
+SIMAP_INITIALIZER(_to_ofport);
+
+/* Contains the list of known tunnels. */
+static struct hmap tunnels = HMAP_INITIALIZER();
+
+/* The last sequence number seen from the southbound IDL. */
+static unsigned int seqno = 0;
+
+void
+force_full_process(void)
+{
+seqno = 0;
+}
+
 int
 main(int argc, char *argv[])
 {
@@ -427,39 +444,51 @@ main(int argc, char *argv[])
 _lports);
 }
 
+enum mf_field_id mff_ovn_geneve;
+bool physical_change = true;
 if (br_int && chassis_id) {
+mff_ovn_geneve = ofctrl_run(br_int);
+physical_change = detect_and_save_physical_changes(
+_to_ofport, , mff_ovn_geneve, br_int,
+chassis_id);
+unsigned int cur_seqno = ovsdb_idl_get_seqno(ovnsb_idl_loop.idl);
+
 patch_run(, br_int, chassis_id, _datapaths,
   _datapaths);
 
 static struct lport_index lports;
-static struct mcgroup_index mcgroups;
 lport_index_init(, ctx.ovnsb_idl);
-mcgroup_index_init(, ctx.ovnsb_idl);
-
-enum mf_field_id mff_ovn_geneve = ofctrl_run(br_int);
-
-pinctrl_run(, , br_int, chassis_id, _datapaths);
+pinctrl_run(, , br_int, chassis_id,
+_datapaths);
 update_ct_zones(_lports, _datapaths, _zones,
 ct_zone_bitmap);
 
-struct hmap flow_table = HMAP_INITIALIZER(_table);
-lflow_run(, , , _datapaths,
-  _datapaths, _table, _zones,
-  _table);
-
-physical_run(, mff_ovn_geneve,
- br_int, chassis_id, _zones, _table,
- _datapaths, _datapaths);
-
-ofctrl_put(_table, get_nb_cfg(ctx.ovnsb_idl));
-hmap_destroy(_table);
-if (ctx.ovnsb_idl_txn) {
-int64_t cur_cfg = ofctrl_get_cur_cfg();
-if (cur_cfg && cur_cfg != chassis->nb_cfg) {
-sbrec_chassis_set_nb_cfg(chassis, cur_cfg);
+
+if (physical_change || seqno < cur_seqno) {
+seqno = cur_seqno;
+
+static struct mcgroup_index mcgroups;
+mcgroup_index_init(, ctx.ovnsb_idl);
+
+struct hmap flow_table = HMAP_INITIALIZER(_table);
+lflow_run(, , , _datapaths,
+  _datapaths, _table, _zones,
+  _table);
+
+physical_run(, mff_ovn_geneve,
+ br_int, chassis_id, _zones, _table,
+ _datapaths, _datapaths);
+
+ofctrl_put(_table, get_nb_cfg(ctx.ovnsb_idl));
+hmap_destroy(_table);
+if (ctx.ovnsb_idl_txn) {
+int64_t cur_cfg = ofctrl_get_cur_cfg();
+if (cur_cfg && cur_cfg != chassis->nb_cfg) {
+

[ovs-dev] [PATCH 1/4] ovn-controller: Back out incremental processing

2016-08-30 Thread Ryan Moats 
As [1] indicates, incremental processing hasn't resulted
in an improvement worth the complexity it has added.
This patch backs out all of the code specific to incremental
processing, along with the persisting of OF flows,
logical ports, multicast groups, all_lports, local and patched
datapaths.

Persisted objects in the ovn/controller/physical.c module will
be used by a future patch set to determine if physical changes
have occurred.

Future patch sets will convert the ovn/controller/encaps.c
module back to full processing and remove the persistance of
address sets in the ovn/controller/lflow.c module.

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 include/ovn/actions.h   |   4 -
 ovn/controller/binding.c| 154 ++
 ovn/controller/binding.h|   1 -
 ovn/controller/encaps.c | 111 ++---
 ovn/controller/lflow.c  | 102 +++-
 ovn/controller/lflow.h  |   4 +-
 ovn/controller/lport.c  | 220 -
 ovn/controller/lport.h  |  24 +--
 ovn/controller/ofctrl.c | 353 +++-
 ovn/controller/ofctrl.h |  16 +-
 ovn/controller/ovn-controller.c |  61 ---
 ovn/controller/patch.c  |   6 -
 ovn/controller/physical.c   | 175 +---
 ovn/controller/physical.h   |   3 +-
 ovn/lib/actions.c   |   1 -
 tests/ovn.at|  55 +++
 tests/system-ovn.at |  45 ++---
 17 files changed, 387 insertions(+), 948 deletions(-)

diff --git a/include/ovn/actions.h b/include/ovn/actions.h
index e2a716a..b678d33 100644
--- a/include/ovn/actions.h
+++ b/include/ovn/actions.h
@@ -298,7 +298,6 @@ struct group_table {
 struct group_info {
 struct hmap_node hmap_node;
 struct ds group;
-struct uuid lflow_uuid;
 uint32_t group_id;
 };
 
@@ -412,9 +411,6 @@ struct ovnact_encode_params {
 /* A struct to figure out the group_id for group actions. */
 struct group_table *group_table;
 
-/* The logical flow uuid that drove this action. */
-struct uuid lflow_uuid;
-
 /* OVN maps each logical flow table (ltable), one-to-one, onto a physical
  * OpenFlow flow table (ptable).  A number of parameters describe this
  * mapping and data related to flow tables:
diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
index c26007d..0353a7b 100644
--- a/ovn/controller/binding.c
+++ b/ovn/controller/binding.c
@@ -30,18 +30,6 @@
 
 VLOG_DEFINE_THIS_MODULE(binding);
 
-/* A set of the iface-id values of local interfaces on this chassis. */
-static struct sset local_ids = SSET_INITIALIZER(_ids);
-
-/* When this gets set to true, the next run will re-check all binding records. 
*/
-static bool process_full_binding = false;
-
-void
-binding_reset_processing(void)
-{
-process_full_binding = true;
-}
-
 void
 binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
 {
@@ -64,16 +52,12 @@ binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
  _interface_col_ingress_policing_burst);
 }
 
-static bool
+static void
 get_local_iface_ids(const struct ovsrec_bridge *br_int,
 struct shash *lport_to_iface,
 struct sset *all_lports)
 {
 int i;
-bool changed = false;
-
-struct sset old_local_ids = SSET_INITIALIZER(_local_ids);
-sset_clone(_local_ids, _ids);
 
 for (i = 0; i < br_int->n_ports; i++) {
 const struct ovsrec_port *port_rec = br_int->ports[i];
@@ -93,53 +77,9 @@ get_local_iface_ids(const struct ovsrec_bridge *br_int,
 continue;
 }
 shash_add(lport_to_iface, iface_id, iface_rec);
-if (!sset_find_and_delete(_local_ids, iface_id)) {
-sset_add(_ids, iface_id);
-sset_add(all_lports, iface_id);
-changed = true;
-}
+sset_add(all_lports, iface_id);
 }
 }
-
-/* Any item left in old_local_ids is an ID for an interface
- * that has been removed. */
-if (!changed && !sset_is_empty(_local_ids)) {
-changed = true;
-
-const char *cur_id;
-SSET_FOR_EACH(cur_id, _local_ids) {
-sset_find_and_delete(_ids, cur_id);
-sset_find_and_delete(all_lports, cur_id);
-}
-}
-
-sset_destroy(_local_ids);
-
-return changed;
-}
-
-static struct local_datapath *
-local_datapath_lookup_by_uuid(struct hmap *hmap_p, const struct uuid *uuid)
-{
-struct local_datapath *ld;
-HMAP_FOR_EACH_WITH_HASH(ld, uuid_hmap_node, uuid_hash(uuid), hmap_p) {
-if (uuid_equals(>uuid, uuid)) {
-return ld;
-}
-}
-return NULL;
-}
-
-static void
-remove_local_datapath(struct hmap *local_datapaths, struct local_datapath *ld)
-{
-if (ld->logical_port) {
-free(ld->logical_port);

[ovs-dev] [PATCH 0/4] ovn-controller: Replace incremental processing with quiet mode

2016-08-30 Thread Ryan Moats 
This patch set removes incremental processing and replaces it
with quiet mode: ovn-controller first tests to see if recalculation
is necessary and if so, does a full calculation.

Side effect of this is that almost all data persistence is also reverted.
The exception is the persistence in the encaps.c module - it's not clear
how to revert that while still maintaining 100% unit test pass.

Note: these patches previously existed as individually versioned patches.
There is no change from v7, v4, v1, and v1 (in order).

Ryan Moats (4):
  ovn-controller: Back out incremental processing
  ovn-controller: add quiet mode
  Unpersist data structures for address sets.
  ovn-controller: Convert encaps module back to full processing

 include/ovn/actions.h   |   4 -
 ovn/controller/binding.c| 154 ++
 ovn/controller/binding.h|   1 -
 ovn/controller/encaps.c | 145 +++--
 ovn/controller/lflow.c  | 258 +++--
 ovn/controller/lflow.h  |   4 +-
 ovn/controller/lport.c  | 220 -
 ovn/controller/lport.h  |  24 +--
 ovn/controller/ofctrl.c | 353 +++-
 ovn/controller/ofctrl.h |  16 +-
 ovn/controller/ovn-controller.c | 106 
 ovn/controller/ovn-controller.h |   1 +
 ovn/controller/patch.c  |  17 +-
 ovn/controller/physical.c   | 263 +++---
 ovn/controller/physical.h   |   8 +-
 ovn/lib/actions.c   |   1 -
 ovn/lib/expr.c  |   1 +
 tests/ovn.at|  55 +++
 tests/system-ovn.at |  45 ++---
 19 files changed, 549 insertions(+), 1127 deletions(-)

-- 
2.7.4

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] patch series

2016-08-30 Thread Ryan Moats 

Ben Pfaff <b...@ovn.org> wrote on 08/30/2016 05:05:14 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/30/2016 05:05 PM
> Subject: patch series
>
> It looks like you have 4 separate patches that are connected and in
> which 3 of them depend on the other one.
>
> Usually, we'd represent that as a 4-patch series.
>

I went back and forth on whether to put them together or not,
and decided to leave the patches that were already there be
and build on top of them, rather than resubmit them with only
a numbering change in the subject.

Given that was the wrong approach, a series will follow shortly...
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] ovn-controller: Convert encaps module back to full processing

2016-08-28 Thread Ryan Moats 


Ryan Moats/Omaha/IBM@IBMUS wrote on 08/28/2016 05:51:38 PM:

> From: Ryan Moats/Omaha/IBM@IBMUS
> To: dev@openvswitch.org
> Cc: b...@ovn.org, Ryan Moats/Omaha/IBM@IBMUS
> Date: 08/28/2016 05:51 PM
> Subject: [PATCH] ovn-controller: Convert encaps module back to full
processing
>
> Commit f5d916cb ("ovn-controller: Back out incremental processing")
> left the conversion of the encaps module to a future patch set.
> This patch converts this module back to full processing, but
> does not remove all persistence of associated data strcutures.
>
> This commit depends on f5d916cb ("ovn-controller:
> Back out incremental processing").
>
> Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> ---

This patch set does not remove persistence because I'm not sure how
to do it and still maintain correct behavior for the "change Encap
properties" unit test case.  In fact, my original attempt was to
just go back to the original code base, but that failed on that
particular test case...
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] Unpersist data structures for address sets.

2016-08-28 Thread Ryan Moats 
Ryan Moats/Omaha/IBM@IBMUS wrote on 08/28/2016 05:51:02 PM:

> From: Ryan Moats/Omaha/IBM@IBMUS
> To: dev@openvswitch.org
> Cc: b...@ovn.org, Ryan Moats/Omaha/IBM@IBMUS
> Date: 08/28/2016 05:51 PM
> Subject: [PATCH] Unpersist data structures for address sets.
>
> With the removal of incremental processing, it is no longer
> necessary to persist the data structures for storing address
> sets.  Simplify things by removing this complexity.
>
> Side effect: fixed a memory leak in expr_macros_destroy that
> is evidenced by this change.
>
> This commit depends on f5d916cb ("ovn-controller:
> Back out incremental processing").
>
> Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> ---

Please disregard the above - I resent it by mistake...
(I've marked the patch "Not Applicable" in patchworks)
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH] Unpersist data structures for address sets.

2016-08-28 Thread Ryan Moats 
With the removal of incremental processing, it is no longer
necessary to persist the data structures for storing address
sets.  Simplify things by removing this complexity.

Side effect: fixed a memory leak in expr_macros_destroy that
is evidenced by this change.

This commit depends on f5d916cb ("ovn-controller:
Back out incremental processing").

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/lflow.c | 166 -
 ovn/lib/expr.c |   1 +
 2 files changed, 42 insertions(+), 125 deletions(-)

diff --git a/ovn/controller/lflow.c b/ovn/controller/lflow.c
index dc69047..5713c46 100644
--- a/ovn/controller/lflow.c
+++ b/ovn/controller/lflow.c
@@ -38,14 +38,10 @@ VLOG_DEFINE_THIS_MODULE(lflow);
 /* Contains "struct expr_symbol"s for fields supported by OVN lflows. */
 static struct shash symtab;
 
-/* Contains an internal expr datastructure that represents an address set. */
-static struct shash expr_address_sets;
-
 void
 lflow_init(void)
 {
 ovn_init_symtab();
-shash_init(_address_sets);
 }
 
 /* Details of an address set currently in address_sets. We keep a cached
@@ -56,54 +52,6 @@ struct address_set {
 size_t n_addresses;
 };
 
-/* struct address_set instances for address sets currently in the symtab,
- * hashed on the address set name. */
-static struct shash local_address_sets = 
SHASH_INITIALIZER(_address_sets);
-
-static int
-addr_cmp(const void *p1, const void *p2)
-{
-const char *s1 = p1;
-const char *s2 = p2;
-return strcmp(s1, s2);
-}
-
-/* Return true if the address sets match, false otherwise. */
-static bool
-address_sets_match(const struct address_set *addr_set,
-   const struct sbrec_address_set *addr_set_rec)
-{
-char **addrs1;
-char **addrs2;
-
-if (addr_set->n_addresses != addr_set_rec->n_addresses) {
-return false;
-}
-size_t n_addresses = addr_set->n_addresses;
-
-addrs1 = xmemdup(addr_set->addresses,
- n_addresses * sizeof addr_set->addresses[0]);
-addrs2 = xmemdup(addr_set_rec->addresses,
- n_addresses * sizeof addr_set_rec->addresses[0]);
-
-qsort(addrs1, n_addresses, sizeof *addrs1, addr_cmp);
-qsort(addrs2, n_addresses, sizeof *addrs2, addr_cmp);
-
-bool res = true;
-size_t i;
-for (i = 0; i <  n_addresses; i++) {
-if (strcmp(addrs1[i], addrs2[i])) {
-res = false;
-break;
-}
-}
-
-free(addrs1);
-free(addrs2);
-
-return res;
-}
-
 static void
 address_set_destroy(struct address_set *addr_set)
 {
@@ -118,79 +66,34 @@ address_set_destroy(struct address_set *addr_set)
 }
 
 static void
-update_address_sets(struct controller_ctx *ctx)
-{
-/* Remember the names of all address sets currently in expr_address_sets
- * so we can detect address sets that have been deleted. */
-struct sset cur_addr_set_names = SSET_INITIALIZER(_addr_set_names);
-
-struct shash_node *node;
-SHASH_FOR_EACH (node, _address_sets) {
-sset_add(_addr_set_names, node->name);
-}
+update_address_sets(struct controller_ctx *ctx,
+struct shash *local_address_sets_p,
+struct shash *expr_address_sets_p)
 
+{
 /* Iterate address sets in the southbound database.  Create and update the
  * corresponding symtab entries as necessary. */
 const struct sbrec_address_set *addr_set_rec;
 SBREC_ADDRESS_SET_FOR_EACH (addr_set_rec, ctx->ovnsb_idl) {
-struct address_set *addr_set =
-shash_find_data(_address_sets, addr_set_rec->name);
-
-bool create_set = false;
-if (addr_set) {
-/* This address set has already been added.  We must determine
- * if the symtab entry needs to be updated due to a change. */
-sset_find_and_delete(_addr_set_names, addr_set_rec->name);
-if (!address_sets_match(addr_set, addr_set_rec)) {
-shash_find_and_delete(_address_sets, addr_set_rec->name);
-expr_macros_remove(_address_sets, addr_set_rec->name);
-address_set_destroy(addr_set);
-addr_set = NULL;
-create_set = true;
+/* Create a symbol that resolves to the full set of addresses.
+ * Store it in address_sets to remember that we created this
+ * symbol. */
+struct address_set *addr_set = xzalloc(sizeof *addr_set);
+addr_set->n_addresses = addr_set_rec->n_addresses;
+if (addr_set_rec->n_addresses) {
+addr_set->addresses = xmalloc(addr_set_rec->n_addresses
+  * sizeof addr_set->addresses[0]);
+size_t i;
+for (i = 0; i < addr_set_rec->n_addresses; i++) {
+addr_set->addresses[i] = xstrdup(addr_set_rec->addresses[i]);

[ovs-dev] [PATCH] ovn-controller: Convert encaps module back to full processing

2016-08-28 Thread Ryan Moats 
Commit f5d916cb ("ovn-controller: Back out incremental processing")
left the conversion of the encaps module to a future patch set.
This patch converts this module back to full processing, but
does not remove all persistence of associated data strcutures.

This commit depends on f5d916cb ("ovn-controller:
Back out incremental processing").

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/encaps.c | 34 +++---
 1 file changed, 15 insertions(+), 19 deletions(-)

diff --git a/ovn/controller/encaps.c b/ovn/controller/encaps.c
index d99ba05..87f 100644
--- a/ovn/controller/encaps.c
+++ b/ovn/controller/encaps.c
@@ -412,25 +412,21 @@ encaps_run(struct controller_ctx *ctx, const struct 
ovsrec_bridge *br_int,
 /* Maintain a mapping backwards from encap entries to their parent
  * chassis. Most changes happen at the encap row entry but tunnels need
  * to be established on the basis of the overall chassis. */
-SBREC_CHASSIS_FOR_EACH_TRACKED (chassis_rec, ctx->ovnsb_idl) {
-/* Defer deletion of mapping until we have cleaned up associated
- * ports. */
-if (!sbrec_chassis_is_deleted(chassis_rec)) {
-for (int i = 0; i < chassis_rec->n_encaps; i++) {
-encap_rec = chassis_rec->encaps[i];
-
-struct encap_hash_node *encap_hash_node;
-encap_hash_node = lookup_encap_uuid(_rec->header_.uuid);
-if (encap_hash_node) {
-/* A change might have invalidated our mapping. Process the
- * new version and then iterate over everything to see if 
it
- * is OK. */
-delete_encap_uuid(encap_hash_node);
-poll_immediate_wake();
-}
-
-insert_encap_uuid(_rec->header_.uuid, chassis_rec);
+SBREC_CHASSIS_FOR_EACH (chassis_rec, ctx->ovnsb_idl) {
+for (int i = 0; i < chassis_rec->n_encaps; i++) {
+encap_rec = chassis_rec->encaps[i];
+
+struct encap_hash_node *encap_hash_node;
+encap_hash_node = lookup_encap_uuid(_rec->header_.uuid);
+if (encap_hash_node) {
+/* A change might have invalidated our mapping. Process the
+ * new version and then iterate over everything to see if it
+ * is OK. */
+delete_encap_uuid(encap_hash_node);
+poll_immediate_wake();
 }
+
+insert_encap_uuid(_rec->header_.uuid, chassis_rec);
 }
 }
 
@@ -440,7 +436,7 @@ encaps_run(struct controller_ctx *ctx, const struct 
ovsrec_bridge *br_int,
  * might actually result in the creation of a different type tunnel if
  * that type is preferred. That's OK - when we process the other encap
  * rows, we'll just skip over the new tunnels. */
-SBREC_ENCAP_FOR_EACH_TRACKED (encap_rec, ctx->ovnsb_idl) {
+SBREC_ENCAP_FOR_EACH (encap_rec, ctx->ovnsb_idl) {
 struct encap_hash_node *encap_hash_node;
 struct chassis_hash_node *chassis_hash_node;
 const struct ovsrec_port *port_rec = NULL;
-- 
2.7.4

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH] Unpersist data structures for address sets.

2016-08-27 Thread Ryan Moats 
With the removal of incremental processing, it is no longer
necessary to persist the data structures for storing address
sets.  Simplify things by removing this complexity.

Side effect: fixed a memory leak in expr_macros_destroy that
is evidenced by this change.

This commit depends on f5d916cb ("ovn-controller:
Back out incremental processing").

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/lflow.c | 166 -
 ovn/lib/expr.c |   1 +
 2 files changed, 42 insertions(+), 125 deletions(-)

diff --git a/ovn/controller/lflow.c b/ovn/controller/lflow.c
index dc69047..5713c46 100644
--- a/ovn/controller/lflow.c
+++ b/ovn/controller/lflow.c
@@ -38,14 +38,10 @@ VLOG_DEFINE_THIS_MODULE(lflow);
 /* Contains "struct expr_symbol"s for fields supported by OVN lflows. */
 static struct shash symtab;
 
-/* Contains an internal expr datastructure that represents an address set. */
-static struct shash expr_address_sets;
-
 void
 lflow_init(void)
 {
 ovn_init_symtab();
-shash_init(_address_sets);
 }
 
 /* Details of an address set currently in address_sets. We keep a cached
@@ -56,54 +52,6 @@ struct address_set {
 size_t n_addresses;
 };
 
-/* struct address_set instances for address sets currently in the symtab,
- * hashed on the address set name. */
-static struct shash local_address_sets = 
SHASH_INITIALIZER(_address_sets);
-
-static int
-addr_cmp(const void *p1, const void *p2)
-{
-const char *s1 = p1;
-const char *s2 = p2;
-return strcmp(s1, s2);
-}
-
-/* Return true if the address sets match, false otherwise. */
-static bool
-address_sets_match(const struct address_set *addr_set,
-   const struct sbrec_address_set *addr_set_rec)
-{
-char **addrs1;
-char **addrs2;
-
-if (addr_set->n_addresses != addr_set_rec->n_addresses) {
-return false;
-}
-size_t n_addresses = addr_set->n_addresses;
-
-addrs1 = xmemdup(addr_set->addresses,
- n_addresses * sizeof addr_set->addresses[0]);
-addrs2 = xmemdup(addr_set_rec->addresses,
- n_addresses * sizeof addr_set_rec->addresses[0]);
-
-qsort(addrs1, n_addresses, sizeof *addrs1, addr_cmp);
-qsort(addrs2, n_addresses, sizeof *addrs2, addr_cmp);
-
-bool res = true;
-size_t i;
-for (i = 0; i <  n_addresses; i++) {
-if (strcmp(addrs1[i], addrs2[i])) {
-res = false;
-break;
-}
-}
-
-free(addrs1);
-free(addrs2);
-
-return res;
-}
-
 static void
 address_set_destroy(struct address_set *addr_set)
 {
@@ -118,79 +66,34 @@ address_set_destroy(struct address_set *addr_set)
 }
 
 static void
-update_address_sets(struct controller_ctx *ctx)
-{
-/* Remember the names of all address sets currently in expr_address_sets
- * so we can detect address sets that have been deleted. */
-struct sset cur_addr_set_names = SSET_INITIALIZER(_addr_set_names);
-
-struct shash_node *node;
-SHASH_FOR_EACH (node, _address_sets) {
-sset_add(_addr_set_names, node->name);
-}
+update_address_sets(struct controller_ctx *ctx,
+struct shash *local_address_sets_p,
+struct shash *expr_address_sets_p)
 
+{
 /* Iterate address sets in the southbound database.  Create and update the
  * corresponding symtab entries as necessary. */
 const struct sbrec_address_set *addr_set_rec;
 SBREC_ADDRESS_SET_FOR_EACH (addr_set_rec, ctx->ovnsb_idl) {
-struct address_set *addr_set =
-shash_find_data(_address_sets, addr_set_rec->name);
-
-bool create_set = false;
-if (addr_set) {
-/* This address set has already been added.  We must determine
- * if the symtab entry needs to be updated due to a change. */
-sset_find_and_delete(_addr_set_names, addr_set_rec->name);
-if (!address_sets_match(addr_set, addr_set_rec)) {
-shash_find_and_delete(_address_sets, addr_set_rec->name);
-expr_macros_remove(_address_sets, addr_set_rec->name);
-address_set_destroy(addr_set);
-addr_set = NULL;
-create_set = true;
+/* Create a symbol that resolves to the full set of addresses.
+ * Store it in address_sets to remember that we created this
+ * symbol. */
+struct address_set *addr_set = xzalloc(sizeof *addr_set);
+addr_set->n_addresses = addr_set_rec->n_addresses;
+if (addr_set_rec->n_addresses) {
+addr_set->addresses = xmalloc(addr_set_rec->n_addresses
+  * sizeof addr_set->addresses[0]);
+size_t i;
+for (i = 0; i < addr_set_rec->n_addresses; i++) {
+addr_set->addresses[i] = xstrdup(addr_set_rec->addresses[i]);

Re: [ovs-dev] [PATCH v3] ovn-controller: add quiet mode

2016-08-27 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 08/27/2016 11:45:57 AM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/27/2016 11:46 AM
> Subject: Re: [PATCH v3] ovn-controller: add quiet mode
>
> On Fri, Aug 26, 2016 at 07:23:22PM -0500, Ryan Moats wrote:
> >
> >
> > Ryan Moats/Omaha/IBM@IBMUS wrote on 08/26/2016 04:36:20 PM:
> >
> > > From: Ryan Moats/Omaha/IBM@IBMUS
> > > To: dev@openvswitch.org
> > > Cc: b...@ovn.org, Ryan Moats/Omaha/IBM@IBMUS
> > > Date: 08/26/2016 04:36 PM
> > > Subject: [PATCH v3] ovn-controller: add quiet mode
> > >
> > > As discussed in [1], what the incremental processing code
> > > actually accomplished was that the ovn-controller would
> > > be "quiet" and not burn CPU when things weren't changing.
> > > This patch set recreates this state by calculating whether
> > > changes have occured that would require a full calculation
> > > to be performed.
> > >
> > > This commit depends on f5d916cb ("ovn-controller:
> > > Back out incremental processing")
> > >
> > > [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
> > >
> > > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> > > ---
> >
> > Not quite sure why this was marked as Not Applicable in patch
> > works, as it sill applies cleanly to v6 of the remove incremental
> > processing patch, so I've put it back as new...
>
> I couldn't get it to apply.  I'll try again.
>

Ok... in the meantime, I realized that I wanted to unpersist some
of the physical.c items in that follow on patch set, so there is a v7
of remove incremental processing (v7 patched a memory leak I made)
and a v4 of the quite mode patch, which should apply cleanly to
v7.  I'm currently working on the "unpersist address sets" follow
on patch and once it clears memory testing, I'll post it...

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v7] ovn-controller: Back out incremental processing

2016-08-27 Thread Ryan Moats 
As [1] indicates, incremental processing hasn't resulted
in an improvement worth the complexity it has added.
This patch backs out all of the code specific to incremental
processing, along with the persisting of OF flows,
logical ports, multicast groups, all_lports, local and patched
datapaths.

Persisted objects in the ovn/controller/physical.c module will
be used by a future patch set to determine if physical changes
have occurred.

Future patch sets will convert the ovn/controller/encaps.c
module back to full processing and remove the persistance of
address sets in the ovn/controller/lflow.c module.

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 v6->v7: corrected a memory leak found during follow-on
   patch testing
 include/ovn/actions.h   |   4 -
 ovn/controller/binding.c| 154 ++
 ovn/controller/binding.h|   1 -
 ovn/controller/encaps.c | 111 ++---
 ovn/controller/lflow.c  | 102 +++-
 ovn/controller/lflow.h  |   4 +-
 ovn/controller/lport.c  | 220 -
 ovn/controller/lport.h  |  24 +--
 ovn/controller/ofctrl.c | 353 +++-
 ovn/controller/ofctrl.h |  16 +-
 ovn/controller/ovn-controller.c |  61 ---
 ovn/controller/patch.c  |   6 -
 ovn/controller/physical.c   | 175 +---
 ovn/controller/physical.h   |   3 +-
 ovn/lib/actions.c   |   1 -
 tests/ovn.at|  55 +++
 tests/system-ovn.at |  45 ++---
 17 files changed, 387 insertions(+), 948 deletions(-)

diff --git a/include/ovn/actions.h b/include/ovn/actions.h
index e2a716a..b678d33 100644
--- a/include/ovn/actions.h
+++ b/include/ovn/actions.h
@@ -298,7 +298,6 @@ struct group_table {
 struct group_info {
 struct hmap_node hmap_node;
 struct ds group;
-struct uuid lflow_uuid;
 uint32_t group_id;
 };
 
@@ -412,9 +411,6 @@ struct ovnact_encode_params {
 /* A struct to figure out the group_id for group actions. */
 struct group_table *group_table;
 
-/* The logical flow uuid that drove this action. */
-struct uuid lflow_uuid;
-
 /* OVN maps each logical flow table (ltable), one-to-one, onto a physical
  * OpenFlow flow table (ptable).  A number of parameters describe this
  * mapping and data related to flow tables:
diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
index c26007d..0353a7b 100644
--- a/ovn/controller/binding.c
+++ b/ovn/controller/binding.c
@@ -30,18 +30,6 @@
 
 VLOG_DEFINE_THIS_MODULE(binding);
 
-/* A set of the iface-id values of local interfaces on this chassis. */
-static struct sset local_ids = SSET_INITIALIZER(_ids);
-
-/* When this gets set to true, the next run will re-check all binding records. 
*/
-static bool process_full_binding = false;
-
-void
-binding_reset_processing(void)
-{
-process_full_binding = true;
-}
-
 void
 binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
 {
@@ -64,16 +52,12 @@ binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
  _interface_col_ingress_policing_burst);
 }
 
-static bool
+static void
 get_local_iface_ids(const struct ovsrec_bridge *br_int,
 struct shash *lport_to_iface,
 struct sset *all_lports)
 {
 int i;
-bool changed = false;
-
-struct sset old_local_ids = SSET_INITIALIZER(_local_ids);
-sset_clone(_local_ids, _ids);
 
 for (i = 0; i < br_int->n_ports; i++) {
 const struct ovsrec_port *port_rec = br_int->ports[i];
@@ -93,53 +77,9 @@ get_local_iface_ids(const struct ovsrec_bridge *br_int,
 continue;
 }
 shash_add(lport_to_iface, iface_id, iface_rec);
-if (!sset_find_and_delete(_local_ids, iface_id)) {
-sset_add(_ids, iface_id);
-sset_add(all_lports, iface_id);
-changed = true;
-}
+sset_add(all_lports, iface_id);
 }
 }
-
-/* Any item left in old_local_ids is an ID for an interface
- * that has been removed. */
-if (!changed && !sset_is_empty(_local_ids)) {
-changed = true;
-
-const char *cur_id;
-SSET_FOR_EACH(cur_id, _local_ids) {
-sset_find_and_delete(_ids, cur_id);
-sset_find_and_delete(all_lports, cur_id);
-}
-}
-
-sset_destroy(_local_ids);
-
-return changed;
-}
-
-static struct local_datapath *
-local_datapath_lookup_by_uuid(struct hmap *hmap_p, const struct uuid *uuid)
-{
-struct local_datapath *ld;
-HMAP_FOR_EACH_WITH_HASH(ld, uuid_hmap_node, uuid_hash(uuid), hmap_p) {
-if (uuid_equals(>uuid, uuid)) {
-return ld;
-}
-}
-return NULL;
-}
-
-static void
-remove_local_datapath(struct hmap *local_datapaths, struct local_datapath *ld)

[ovs-dev] [PATCH v4] ovn-controller: add quiet mode

2016-08-26 Thread Ryan Moats 
As discussed in [1], what the incremental processing code
actually accomplished was that the ovn-controller would
be "quiet" and not burn CPU when things weren't changing.
This patch set recreates this state by calculating whether
changes have occured that would require a full calculation
to be performed.  It does this by persisting a copy of
the localvif_to_ofport and tunnel information in the
controller module, rather than in the physical.c module
as was the case with previous commits.

This commit depends on f5d916cb ("ovn-controller:
Back out incremental processing").

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/ofctrl.c |   2 +
 ovn/controller/ovn-controller.c |  73 -
 ovn/controller/ovn-controller.h |   1 +
 ovn/controller/patch.c  |  11 +
 ovn/controller/physical.c   | 100 
 ovn/controller/physical.h   |   5 ++
 6 files changed, 131 insertions(+), 61 deletions(-)

diff --git a/ovn/controller/ofctrl.c b/ovn/controller/ofctrl.c
index d8e111d..5b1434e 100644
--- a/ovn/controller/ofctrl.c
+++ b/ovn/controller/ofctrl.c
@@ -366,6 +366,8 @@ run_S_CLEAR_FLOWS(void)
 queue_msg(encode_group_mod());
 ofputil_uninit_group_mod();
 
+force_full_process();
+
 /* Clear existing groups, to match the state of the switch. */
 if (groups) {
 ovn_group_table_clear(groups, true);
diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c
index 41e0eb0..34a19ae 100644
--- a/ovn/controller/ovn-controller.c
+++ b/ovn/controller/ovn-controller.c
@@ -304,6 +304,23 @@ get_nb_cfg(struct ovsdb_idl *idl)
 return sb ? sb->nb_cfg : 0;
 }
 
+/* Contains mapping of localvifs to OF ports for detecting if the physical
+ * configuration of the switch has changed. */
+static struct simap localvif_to_ofport =
+SIMAP_INITIALIZER(_to_ofport);
+
+/* Contains the list of known tunnels. */
+static struct hmap tunnels = HMAP_INITIALIZER();
+
+/* The last sequence number seen from the southbound IDL. */
+static unsigned int seqno = 0;
+
+void
+force_full_process(void)
+{
+seqno = 0;
+}
+
 int
 main(int argc, char *argv[])
 {
@@ -427,39 +444,51 @@ main(int argc, char *argv[])
 _lports);
 }
 
+enum mf_field_id mff_ovn_geneve;
+bool physical_change = true;
 if (br_int && chassis_id) {
+mff_ovn_geneve = ofctrl_run(br_int);
+physical_change = detect_and_save_physical_changes(
+_to_ofport, , mff_ovn_geneve, br_int,
+chassis_id);
+unsigned int cur_seqno = ovsdb_idl_get_seqno(ovnsb_idl_loop.idl);
+
 patch_run(, br_int, chassis_id, _datapaths,
   _datapaths);
 
 static struct lport_index lports;
-static struct mcgroup_index mcgroups;
 lport_index_init(, ctx.ovnsb_idl);
-mcgroup_index_init(, ctx.ovnsb_idl);
-
-enum mf_field_id mff_ovn_geneve = ofctrl_run(br_int);
-
-pinctrl_run(, , br_int, chassis_id, _datapaths);
+pinctrl_run(, , br_int, chassis_id,
+_datapaths);
 update_ct_zones(_lports, _datapaths, _zones,
 ct_zone_bitmap);
 
-struct hmap flow_table = HMAP_INITIALIZER(_table);
-lflow_run(, , , _datapaths,
-  _datapaths, _table, _zones,
-  _table);
-
-physical_run(, mff_ovn_geneve,
- br_int, chassis_id, _zones, _table,
- _datapaths, _datapaths);
-
-ofctrl_put(_table, get_nb_cfg(ctx.ovnsb_idl));
-hmap_destroy(_table);
-if (ctx.ovnsb_idl_txn) {
-int64_t cur_cfg = ofctrl_get_cur_cfg();
-if (cur_cfg && cur_cfg != chassis->nb_cfg) {
-sbrec_chassis_set_nb_cfg(chassis, cur_cfg);
+
+if (physical_change || seqno < cur_seqno) {
+seqno = cur_seqno;
+
+static struct mcgroup_index mcgroups;
+mcgroup_index_init(, ctx.ovnsb_idl);
+
+struct hmap flow_table = HMAP_INITIALIZER(_table);
+lflow_run(, , , _datapaths,
+  _datapaths, _table, _zones,
+  _table);
+
+physical_run(, mff_ovn_geneve,
+ br_int, chassis_id, _zones, _table,
+ _datapaths, _datapaths);
+
+ofctrl_put(_table, get_nb_cfg(ctx.ovnsb_idl));
+hmap_destroy(_table);
+if (ctx.ovnsb_idl_txn) {
+int64_t cur_cfg = ofctrl_get_cur_cfg();
+if (cur_cfg && cur_cfg != chassis->nb_cfg) {
+

Re: [ovs-dev] [PATCH v3] ovn-controller: add quiet mode

2016-08-26 Thread Ryan Moats 


Ryan Moats/Omaha/IBM@IBMUS wrote on 08/26/2016 04:36:20 PM:

> From: Ryan Moats/Omaha/IBM@IBMUS
> To: dev@openvswitch.org
> Cc: b...@ovn.org, Ryan Moats/Omaha/IBM@IBMUS
> Date: 08/26/2016 04:36 PM
> Subject: [PATCH v3] ovn-controller: add quiet mode
>
> As discussed in [1], what the incremental processing code
> actually accomplished was that the ovn-controller would
> be "quiet" and not burn CPU when things weren't changing.
> This patch set recreates this state by calculating whether
> changes have occured that would require a full calculation
> to be performed.
>
> This commit depends on f5d916cb ("ovn-controller:
> Back out incremental processing")
>
> [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
>
> Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> ---

Not quite sure why this was marked as Not Applicable in patch
works, as it sill applies cleanly to v6 of the remove incremental
processing patch, so I've put it back as new...

Ryan
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v6] ovn-controller: Back out incremental processing

2016-08-26 Thread Ryan Moats 
As [1] indicates, incremental processing hasn't resulted
in an improvement worth the complexity it has added.
This patch backs out all of the code specific to incremental
processing, along with the persisting of OF flows,
logical ports, multicast groups, all_lports, local and patched
datapaths.

Persisted objects in the ovn/controller/physical.c module will
be used by a future patch set to determine if physical changes
have occurred.

Future patch sets will convert the ovn/controller/encaps.c
module back to full processing and remove the persistance of
address sets in the ovn/controller/lflow.c module.

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 include/ovn/actions.h   |   4 -
 ovn/controller/binding.c| 154 ++
 ovn/controller/binding.h|   1 -
 ovn/controller/encaps.c | 111 ++---
 ovn/controller/lflow.c  | 102 +++-
 ovn/controller/lflow.h  |   4 +-
 ovn/controller/lport.c  | 220 -
 ovn/controller/lport.h  |  24 +--
 ovn/controller/ofctrl.c | 353 +++-
 ovn/controller/ofctrl.h |  16 +-
 ovn/controller/ovn-controller.c |  60 ---
 ovn/controller/patch.c  |   6 -
 ovn/controller/physical.c   | 175 +---
 ovn/controller/physical.h   |   3 +-
 ovn/lib/actions.c   |   1 -
 tests/ovn.at|  55 +++
 tests/system-ovn.at |  45 ++---
 17 files changed, 386 insertions(+), 948 deletions(-)

diff --git a/include/ovn/actions.h b/include/ovn/actions.h
index e2a716a..b678d33 100644
--- a/include/ovn/actions.h
+++ b/include/ovn/actions.h
@@ -298,7 +298,6 @@ struct group_table {
 struct group_info {
 struct hmap_node hmap_node;
 struct ds group;
-struct uuid lflow_uuid;
 uint32_t group_id;
 };
 
@@ -412,9 +411,6 @@ struct ovnact_encode_params {
 /* A struct to figure out the group_id for group actions. */
 struct group_table *group_table;
 
-/* The logical flow uuid that drove this action. */
-struct uuid lflow_uuid;
-
 /* OVN maps each logical flow table (ltable), one-to-one, onto a physical
  * OpenFlow flow table (ptable).  A number of parameters describe this
  * mapping and data related to flow tables:
diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
index c26007d..0353a7b 100644
--- a/ovn/controller/binding.c
+++ b/ovn/controller/binding.c
@@ -30,18 +30,6 @@
 
 VLOG_DEFINE_THIS_MODULE(binding);
 
-/* A set of the iface-id values of local interfaces on this chassis. */
-static struct sset local_ids = SSET_INITIALIZER(_ids);
-
-/* When this gets set to true, the next run will re-check all binding records. 
*/
-static bool process_full_binding = false;
-
-void
-binding_reset_processing(void)
-{
-process_full_binding = true;
-}
-
 void
 binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
 {
@@ -64,16 +52,12 @@ binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
  _interface_col_ingress_policing_burst);
 }
 
-static bool
+static void
 get_local_iface_ids(const struct ovsrec_bridge *br_int,
 struct shash *lport_to_iface,
 struct sset *all_lports)
 {
 int i;
-bool changed = false;
-
-struct sset old_local_ids = SSET_INITIALIZER(_local_ids);
-sset_clone(_local_ids, _ids);
 
 for (i = 0; i < br_int->n_ports; i++) {
 const struct ovsrec_port *port_rec = br_int->ports[i];
@@ -93,53 +77,9 @@ get_local_iface_ids(const struct ovsrec_bridge *br_int,
 continue;
 }
 shash_add(lport_to_iface, iface_id, iface_rec);
-if (!sset_find_and_delete(_local_ids, iface_id)) {
-sset_add(_ids, iface_id);
-sset_add(all_lports, iface_id);
-changed = true;
-}
+sset_add(all_lports, iface_id);
 }
 }
-
-/* Any item left in old_local_ids is an ID for an interface
- * that has been removed. */
-if (!changed && !sset_is_empty(_local_ids)) {
-changed = true;
-
-const char *cur_id;
-SSET_FOR_EACH(cur_id, _local_ids) {
-sset_find_and_delete(_ids, cur_id);
-sset_find_and_delete(all_lports, cur_id);
-}
-}
-
-sset_destroy(_local_ids);
-
-return changed;
-}
-
-static struct local_datapath *
-local_datapath_lookup_by_uuid(struct hmap *hmap_p, const struct uuid *uuid)
-{
-struct local_datapath *ld;
-HMAP_FOR_EACH_WITH_HASH(ld, uuid_hmap_node, uuid_hash(uuid), hmap_p) {
-if (uuid_equals(>uuid, uuid)) {
-return ld;
-}
-}
-return NULL;
-}
-
-static void
-remove_local_datapath(struct hmap *local_datapaths, struct local_datapath *ld)
-{
-if (ld->logical_port) {
-free(ld->logical_port);

Re: [ovs-dev] [ovs-dev, v5] ovn-controller: Back out incremental processing

2016-08-26 Thread Ryan Moats 


Ben Pfaff <b...@ovn.org> wrote on 08/26/2016 06:23:57 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/26/2016 06:24 PM
> Subject: Re: [ovs-dev,v5] ovn-controller: Back out incremental processing
>
> On Fri, Aug 26, 2016 at 04:15:47PM -0500, Ryan Moats wrote:
> >
> >
> > Ben Pfaff <b...@ovn.org> wrote on 08/26/2016 03:27:56 PM:
> >
> > > From: Ben Pfaff <b...@ovn.org>
> > > To: Ryan Moats/Omaha/IBM@IBMUS
> > > Cc: dev@openvswitch.org
> > > Date: 08/26/2016 03:28 PM
> > > Subject: Re: [ovs-dev,v5] ovn-controller: Back out incremental
processing
> > >
> > > On Fri, Aug 26, 2016 at 01:30:59PM +, Ryan Moats wrote:
> > > > As [1] indicates, incremental processing hasn't resulted
> > > > in an improvement worth the complexity it has added.
> > > > This patch backs out all of the code specific to incremental
> > > > processing, along with the persisting of OF flows,
> > > > logical ports, multicast groups, all_lports, local and patched
> > > > datapaths
> > > >
> > > > [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
> > > >
> > > > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> > >
> > > Thanks for working on this.  I think that it is the best approach for
> > > now.  I'm working on a few ideas about ways to build systematic and
> > > reliable incremental update, but they won't be ready for 2.6.
> > >
> > > There is still some incremental behavior in a few places.  I am not
sure
> > > whether this is intentional or if you just missed them.  Can you
comment
> > > on these?
> > >
> > >   - Remaining _FOR_EACH_TRACKED usages in encaps_run().  Actually
it's a
> > > little surprising that encaps still maintains static state,
should
> > > it?
> >
> > I purposely left this file alone because of I haven't been 100%
> > sure how to correctly back out the changes from commit 9263ceaf
> > ("ovn-controller: Make encap processing more robust against changes.").
> > Coupling that with the fact that the problems that have been reported
> > have *not* been with this file since that patch set, I made the
decision
> > to leave it alone for this patch set and address it in a future patch
set.
> >
> > >   - lflow.c has a persistent collection of address sets.
> >
> > I view persisting data as a necessary step towards running
incrementally,
> > but I don't tie the two together (I can run full computations on
persisted
> > data so long as the persistence is correct).  The reason for backing
out
> > the persistence that was backed out was the persistence isn't correct.
> > In the case of address sets, we've not see any errors in our testing,
nor
> > have there been any reported on the mailing list, so I left it as is in
> > this patch set and revisit unpersisting it in a follow-on patch set.
> >
> > >   - physical.c persists lots of state.
> >
> > This one was just a goof - I thought I had unpersisted state here in v4
> > of the patch set but obviously that piece of the change got lost
somewhere
> > along the line...
> >
> > I admit that I should have been more clear about what was going on in
> > binding.c and lflow.c in the commit message and I apologize for that
> > mistake.
> >
> > My question to you is: do you want a v6 of this patch that includes
> > unpersisting things in lflow.c and physical.c or can those show up
> > as a follow-on patch set (I still plan on leaving undoing binding.c
> > to a follow-on patch set).
>
> I'd like a v6 with whatever was missing from physical.c, but we can take
> the others separately.  I wanted to make sure that these were
> intentional and not overlooked.
>

v6 is coming shortly - what is left persisted in physical.c is needed for
the follow on "add quiet mode" patch.
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v3] ovn-controller: add quiet mode

2016-08-26 Thread Ryan Moats 
As discussed in [1], what the incremental processing code
actually accomplished was that the ovn-controller would
be "quiet" and not burn CPU when things weren't changing.
This patch set recreates this state by calculating whether
changes have occured that would require a full calculation
to be performed.

This commit depends on f5d916cb ("ovn-controller:
Back out incremental processing")

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/ofctrl.c |  2 ++
 ovn/controller/ovn-controller.c | 77 +
 ovn/controller/ovn-controller.h |  1 +
 ovn/controller/patch.c  | 11 ++
 ovn/controller/physical.c   | 55 +++--
 ovn/controller/physical.h   |  5 +++
 6 files changed, 110 insertions(+), 41 deletions(-)

diff --git a/ovn/controller/ofctrl.c b/ovn/controller/ofctrl.c
index d8e111d..5b1434e 100644
--- a/ovn/controller/ofctrl.c
+++ b/ovn/controller/ofctrl.c
@@ -366,6 +366,8 @@ run_S_CLEAR_FLOWS(void)
 queue_msg(encode_group_mod());
 ofputil_uninit_group_mod();
 
+force_full_process();
+
 /* Clear existing groups, to match the state of the switch. */
 if (groups) {
 ovn_group_table_clear(groups, true);
diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c
index 41e0eb0..ef424e1 100644
--- a/ovn/controller/ovn-controller.c
+++ b/ovn/controller/ovn-controller.c
@@ -304,6 +304,27 @@ get_nb_cfg(struct ovsdb_idl *idl)
 return sb ? sb->nb_cfg : 0;
 }
 
+/* Contains mapping of localvifs to OF ports for detecting if the physical
+ * configuration of the switch has changed.  This is kept independently of
+ * the same variable name in the physical module as the two are used for
+ * different purposes. */
+static struct simap localvif_to_ofport =
+SIMAP_INITIALIZER(_to_ofport);
+
+/* Contains the list of known tunnels.  This is kept independently of
+ * the same variable name in the physical module as the two are used for
+ * different purposes. */
+static struct hmap tunnels = HMAP_INITIALIZER();
+
+/* The last sequence number seen from the southbound IDL. */
+static unsigned int seqno = 0;
+
+void
+force_full_process(void)
+{
+seqno = 0;
+}
+
 int
 main(int argc, char *argv[])
 {
@@ -427,39 +448,51 @@ main(int argc, char *argv[])
 _lports);
 }
 
+enum mf_field_id mff_ovn_geneve;
+bool physical_change = true;
 if (br_int && chassis_id) {
+mff_ovn_geneve = ofctrl_run(br_int);
+physical_change = detect_and_save_physical_changes(
+_to_ofport, , mff_ovn_geneve, br_int,
+chassis_id);
+unsigned int cur_seqno = ovsdb_idl_get_seqno(ovnsb_idl_loop.idl);
+
 patch_run(, br_int, chassis_id, _datapaths,
   _datapaths);
 
 static struct lport_index lports;
-static struct mcgroup_index mcgroups;
 lport_index_init(, ctx.ovnsb_idl);
-mcgroup_index_init(, ctx.ovnsb_idl);
-
-enum mf_field_id mff_ovn_geneve = ofctrl_run(br_int);
-
-pinctrl_run(, , br_int, chassis_id, _datapaths);
+pinctrl_run(, , br_int, chassis_id,
+_datapaths);
 update_ct_zones(_lports, _datapaths, _zones,
 ct_zone_bitmap);
 
-struct hmap flow_table = HMAP_INITIALIZER(_table);
-lflow_run(, , , _datapaths,
-  _datapaths, _table, _zones,
-  _table);
-
-physical_run(, mff_ovn_geneve,
- br_int, chassis_id, _zones, _table,
- _datapaths, _datapaths);
-
-ofctrl_put(_table, get_nb_cfg(ctx.ovnsb_idl));
-hmap_destroy(_table);
-if (ctx.ovnsb_idl_txn) {
-int64_t cur_cfg = ofctrl_get_cur_cfg();
-if (cur_cfg && cur_cfg != chassis->nb_cfg) {
-sbrec_chassis_set_nb_cfg(chassis, cur_cfg);
+
+if (physical_change || seqno < cur_seqno) {
+seqno = cur_seqno;
+
+static struct mcgroup_index mcgroups;
+mcgroup_index_init(, ctx.ovnsb_idl);
+
+struct hmap flow_table = HMAP_INITIALIZER(_table);
+lflow_run(, , , _datapaths,
+  _datapaths, _table, _zones,
+  _table);
+
+physical_run(, mff_ovn_geneve,
+ br_int, chassis_id, _zones, _table,
+ _datapaths, _datapaths);
+
+ofctrl_put(_table, get_nb_cfg(ctx.ovnsb_idl));
+hmap_destroy(_table);
+if (ctx.ovnsb_idl_txn) {
+int64_t cur_cfg = ofctrl_get_cur_cfg();
+

Re: [ovs-dev] [ovs-dev,v2] ovn-controller: add quiet mode

2016-08-26 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 08/26/2016 03:37:07 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/26/2016 03:37 PM
> Subject: Re: [ovs-dev,v2] ovn-controller: add quiet mode
>
> On Mon, Aug 22, 2016 at 03:25:05AM +, Ryan Moats wrote:
> > As discussed in [1], what the incremental processing code
> > actually accomplished was that the ovn-controller would
> > be "quiet" and not burn CPU when things weren't changing.
> > This patch set recreates this state by calculating whether
> > changes have occured that would require a full calculation
> > to be performed.
> >
> > This commit depends on f5d916cb ("ovn-controller:
> > Back out incremental processing")
> >
> > [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
> >
> > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
>
> I'm getting patch rejects with current master and v5 of the "back out"
> patch.
>

I know that the changes for this rebased onto v5 of the "back out"
branch cleanly.  It's easy enough for me format a patch set for what
I've got now and throw a v3 out there, so I'll do that

Ryan
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [ovs-dev, v5] ovn-controller: Back out incremental processing

2016-08-26 Thread Ryan Moats 


Ben Pfaff <b...@ovn.org> wrote on 08/26/2016 03:27:56 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/26/2016 03:28 PM
> Subject: Re: [ovs-dev,v5] ovn-controller: Back out incremental processing
>
> On Fri, Aug 26, 2016 at 01:30:59PM +, Ryan Moats wrote:
> > As [1] indicates, incremental processing hasn't resulted
> > in an improvement worth the complexity it has added.
> > This patch backs out all of the code specific to incremental
> > processing, along with the persisting of OF flows,
> > logical ports, multicast groups, all_lports, local and patched
> > datapaths
> >
> > [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
> >
> > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
>
> Thanks for working on this.  I think that it is the best approach for
> now.  I'm working on a few ideas about ways to build systematic and
> reliable incremental update, but they won't be ready for 2.6.
>
> There is still some incremental behavior in a few places.  I am not sure
> whether this is intentional or if you just missed them.  Can you comment
> on these?
>
>   - Remaining _FOR_EACH_TRACKED usages in encaps_run().  Actually it's a
> little surprising that encaps still maintains static state, should
> it?

I purposely left this file alone because of I haven't been 100%
sure how to correctly back out the changes from commit 9263ceaf
("ovn-controller: Make encap processing more robust against changes.").
Coupling that with the fact that the problems that have been reported
have *not* been with this file since that patch set, I made the decision
to leave it alone for this patch set and address it in a future patch set.

>   - lflow.c has a persistent collection of address sets.

I view persisting data as a necessary step towards running incrementally,
but I don't tie the two together (I can run full computations on persisted
data so long as the persistence is correct).  The reason for backing out
the persistence that was backed out was the persistence isn't correct.
In the case of address sets, we've not see any errors in our testing, nor
have there been any reported on the mailing list, so I left it as is in
this patch set and revisit unpersisting it in a follow-on patch set.

>   - physical.c persists lots of state.

This one was just a goof - I thought I had unpersisted state here in v4
of the patch set but obviously that piece of the change got lost somewhere
along the line...

I admit that I should have been more clear about what was going on in
binding.c and lflow.c in the commit message and I apologize for that
mistake.

My question to you is: do you want a v6 of this patch that includes
unpersisting things in lflow.c and physical.c or can those show up
as a follow-on patch set (I still plan on leaving undoing binding.c
to a follow-on patch set).

Ryan
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v5] ovn-controller: Back out incremental processing

2016-08-26 Thread Ryan Moats 
As [1] indicates, incremental processing hasn't resulted
in an improvement worth the complexity it has added.
This patch backs out all of the code specific to incremental
processing, along with the persisting of OF flows,
logical ports, multicast groups, all_lports, local and patched
datapaths

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 v1->v2:
 - removed some obvious memory leaks left behind in physical.c
 v2->v3:
 - removed some less obvious memory leaks created by
   non-persisting ofctrl.c 
 v3->v4:
 - unpersist all_lports, local_datapath, patch_datapath structures
 - added initial test courtesy of shettyg
 v4->v5:
 - included review comments from shettyg
 include/ovn/actions.h   |   4 -
 ovn/controller/binding.c| 154 ++
 ovn/controller/binding.h|   1 -
 ovn/controller/encaps.c | 111 ++---
 ovn/controller/lflow.c  | 102 +++-
 ovn/controller/lflow.h  |   4 +-
 ovn/controller/lport.c  | 220 -
 ovn/controller/lport.h  |  24 +--
 ovn/controller/ofctrl.c | 353 +++-
 ovn/controller/ofctrl.h |  16 +-
 ovn/controller/ovn-controller.c |  60 ---
 ovn/controller/patch.c  |   6 -
 ovn/controller/physical.c   | 166 +--
 ovn/controller/physical.h   |   3 +-
 ovn/lib/actions.c   |   1 -
 tests/ovn.at|  55 +++
 tests/system-ovn.at |  45 ++---
 17 files changed, 386 insertions(+), 939 deletions(-)

diff --git a/include/ovn/actions.h b/include/ovn/actions.h
index e2a716a..b678d33 100644
--- a/include/ovn/actions.h
+++ b/include/ovn/actions.h
@@ -298,7 +298,6 @@ struct group_table {
 struct group_info {
 struct hmap_node hmap_node;
 struct ds group;
-struct uuid lflow_uuid;
 uint32_t group_id;
 };
 
@@ -412,9 +411,6 @@ struct ovnact_encode_params {
 /* A struct to figure out the group_id for group actions. */
 struct group_table *group_table;
 
-/* The logical flow uuid that drove this action. */
-struct uuid lflow_uuid;
-
 /* OVN maps each logical flow table (ltable), one-to-one, onto a physical
  * OpenFlow flow table (ptable).  A number of parameters describe this
  * mapping and data related to flow tables:
diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
index c26007d..0353a7b 100644
--- a/ovn/controller/binding.c
+++ b/ovn/controller/binding.c
@@ -30,18 +30,6 @@
 
 VLOG_DEFINE_THIS_MODULE(binding);
 
-/* A set of the iface-id values of local interfaces on this chassis. */
-static struct sset local_ids = SSET_INITIALIZER(_ids);
-
-/* When this gets set to true, the next run will re-check all binding records. 
*/
-static bool process_full_binding = false;
-
-void
-binding_reset_processing(void)
-{
-process_full_binding = true;
-}
-
 void
 binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
 {
@@ -64,16 +52,12 @@ binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
  _interface_col_ingress_policing_burst);
 }
 
-static bool
+static void
 get_local_iface_ids(const struct ovsrec_bridge *br_int,
 struct shash *lport_to_iface,
 struct sset *all_lports)
 {
 int i;
-bool changed = false;
-
-struct sset old_local_ids = SSET_INITIALIZER(_local_ids);
-sset_clone(_local_ids, _ids);
 
 for (i = 0; i < br_int->n_ports; i++) {
 const struct ovsrec_port *port_rec = br_int->ports[i];
@@ -93,53 +77,9 @@ get_local_iface_ids(const struct ovsrec_bridge *br_int,
 continue;
 }
 shash_add(lport_to_iface, iface_id, iface_rec);
-if (!sset_find_and_delete(_local_ids, iface_id)) {
-sset_add(_ids, iface_id);
-sset_add(all_lports, iface_id);
-changed = true;
-}
+sset_add(all_lports, iface_id);
 }
 }
-
-/* Any item left in old_local_ids is an ID for an interface
- * that has been removed. */
-if (!changed && !sset_is_empty(_local_ids)) {
-changed = true;
-
-const char *cur_id;
-SSET_FOR_EACH(cur_id, _local_ids) {
-sset_find_and_delete(_ids, cur_id);
-sset_find_and_delete(all_lports, cur_id);
-}
-}
-
-sset_destroy(_local_ids);
-
-return changed;
-}
-
-static struct local_datapath *
-local_datapath_lookup_by_uuid(struct hmap *hmap_p, const struct uuid *uuid)
-{
-struct local_datapath *ld;
-HMAP_FOR_EACH_WITH_HASH(ld, uuid_hmap_node, uuid_hash(uuid), hmap_p) {
-if (uuid_equals(>uuid, uuid)) {
-return ld;
-}
-}
-return NULL;
-}
-
-static void
-remove_local_datapath(struct hmap *local_datapaths, struct local_datapath *ld)
-{
-if (ld->logical_port) {
-free(ld->log

Re: [ovs-dev] ovsdb-server sb db restart causes ovn-controller to crash

2016-08-25 Thread Ryan Moats 
"dev" <dev-boun...@openvswitch.org> wrote on 08/25/2016 02:15:47 PM:

> From: Richard Theis/Rochester/IBM@IBMUS
> To: dev@openvswitch.org
> Date: 08/25/2016 02:16 PM
> Subject: [ovs-dev] ovsdb-server sb db restart causes ovn-controller to
crash
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> Bug 1604064 (https://bugs.launchpad.net/neutron/+bug/1604064) was
reported
> against
> the networking-ovn project. While investigating, I determined that
> whenever the
> ovsdb-server SB database process is restarted, all ovn-controller
> processes crash.
> I recreated this problem using both an OpenStack DevStack and Vagrant
> deployment
> with networking-ovn.
>
> I briefly spoke with Ryan Moats about this problem and he plans to
> investigate
> further.
>
> - Richard Theis

My analysis is that this is due to the local_datapath structure becoming
stale because it is not flushed after the SB db reconnects.  Because of
this, I tested with v4 of the undo incremental processing patch (which
goes back to recalculating local_datapath each cycle) and I don't see
the same corruption problem when the SB db restarts and ovn-controller
reconnects. Given this, should a separate patch set be spun to address
this problem?

Ryan
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH v4] ovn-controller: Back out incremental processing

2016-08-25 Thread Ryan Moats 



Guru Shetty <g...@ovn.org> wrote on 08/25/2016 03:11:42 PM:

> From: Guru Shetty <g...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: ovs dev <dev@openvswitch.org>
> Date: 08/25/2016 03:11 PM
> Subject: Re: [ovs-dev] [PATCH v4] ovn-controller: Back out
> incremental processing
>
> On 25 August 2016 at 13:04, Guru Shetty <g...@ovn.org> wrote:
>
> On 24 August 2016 at 18:30, Ryan Moats <rmo...@us.ibm.com> wrote:
> As [1] indicates, incremental processing hasn't resulted
> in an improvement worth the complexity it has added.
> This patch backs out all of the code specific to incremental
> processing, along with the persisting of OF flows,
> logical ports, multicast groups, all_lports, local and patched
> datapaths
>
> [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
>
> Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> Co-authored-by: Guru Shetty <g...@ovn.com>
> I think you should remove the co-authored-by. I just gave a very
> silly test that I don't think warrants a co-authored-by.
>
> I think you should add the following incremental to the test to make
> it a little better.
>
> diff --git a/tests/ovn.at b/tests/ovn.at
> index 30411d8..c161d07 100644
> --- a/tests/ovn.at
> +++ b/tests/ovn.at
> @@ -5015,7 +5015,7 @@ for i in `seq 1 3`; do
>      -- lsp-set-addresses bar$i "f0:00:0a:01:02:$i 172.16.1.$ip"
>  done
>
> -OVS_WAIT_UNTIL([ovs-ofctl dump-flows br-int table=0 | grep REG13])
> +OVS_WAIT_UNTIL([test `ovs-ofctl dump-flows br-int table=0 | grep
> REG13 | wc -l` -eq 4])
>
>  OVN_CLEANUP([hv1])
>
> As expected, the load-balancer tests now fail. But in addition to
> the zone allocation, there was another issue which was added as part
> of incremental processing fixes that was causing consistent test
> failures. The following incremental reverts it back to how it was
> before all the incremental processing changes (There is still a
> corner case, but I don't think it was caused by incremental
> processing, so I will send a separate fix for that).
>
> Ignore the incremental below this in the previous mail. It got cut.
> I am pasting a fresh one here:
>
> diff --git a/ovn/controller/lflow.c b/ovn/controller/lflow.c
> index 9fc8a93..647b4f0 100644
> --- a/ovn/controller/lflow.c
> +++ b/ovn/controller/lflow.c
> @@ -253,8 +253,6 @@ add_logical_flows(struct controller_ctx *ctx,
> const struct lport_index *lports,
>      uint32_t conj_id_ofs = 1;
>      const struct sbrec_logical_flow *lflow;
>
> -    ovn_group_table_clear(group_table, false);
> -
>      struct hmap dhcp_opts = HMAP_INITIALIZER(_opts);
>      struct hmap dhcpv6_opts = HMAP_INITIALIZER(_opts);
>      const struct sbrec_dhcp_options *dhcp_opt_row;
> diff --git a/ovn/controller/ofctrl.c b/ovn/controller/ofctrl.c
> index 5c3125a..d8e111d 100644
> --- a/ovn/controller/ofctrl.c
> +++ b/ovn/controller/ofctrl.c
> @@ -708,16 +708,6 @@ add_flow_mod(struct ofputil_flow_mod *fm,
> struct ovs_list *msgs)
>  ^L
>  /* group_table. */
>
> -static struct group_info *
> -group_info_clone(struct group_info *source)
> -{
> -    struct group_info *clone = xmalloc(sizeof *clone);
> -    ds_clone(>group, >group);
> -    clone->group_id = source->group_id;
> -    clone->hmap_node.hash = source->hmap_node.hash;
> -    return clone;
> -}
> -
>  /* Finds and returns a group_info in 'existing_groups' whose key
isidentical
>   * to 'target''s key, or NULL if there is none. */
>  static struct group_info *
> @@ -770,7 +760,10 @@ add_group_mod(const struct ofputil_group_mod
> *gm, struct ovs_list *msgs)
>   * with ofctrl_add_flow().
>   *
>   * Replaces the group table on the switch, if possible, by the contents
of
> - * 'groups->desired_groups'.
> + * 'groups->desired_groups'.  Regardless of whether the group table
> + * is updated, this deletes all the groups from the
> + * 'groups->desired_groups' and frees them. (The hmap itself isn't
> + * destroyed.)
>   *
>   * This should be called after ofctrl_run() within the main loop. */
>  void
> @@ -929,10 +922,13 @@ ofctrl_put(struct hmap *flow_table, int64_t nb_cfg)
>      /* Move the contents of desired_groups to existing_groups. */
>      HMAP_FOR_EACH_SAFE(desired, next_group, hmap_node,
>                         >desired_groups) {
> +        hmap_remove(>desired_groups, >hmap_node);
>          if (!ovn_group_lookup(>existing_groups, desired)) {
> -            struct group_info *clone = group_info_clone(desired);
> -            hmap_insert(>existing_groups, >hmap_node,
> -                        clone->hmap_node.hash);
> +            hmap_insert(>existing_groups, >hmap_no

Re: [ovs-dev] [PATCH v4] ovn-controller: Back out incremental processing

2016-08-25 Thread Ryan Moats 

Russell Bryant <russ...@ovn.org> wrote on 08/25/2016 12:45:44 PM:

> From: Russell Bryant <russ...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: ovs dev <dev@openvswitch.org>
> Date: 08/25/2016 12:46 PM
> Subject: Re: [ovs-dev] [PATCH v4] ovn-controller: Back out
> incremental processing
>
> On Wed, Aug 24, 2016 at 9:30 PM, Ryan Moats <rmo...@us.ibm.com> wrote:
> As [1] indicates, incremental processing hasn't resulted
> in an improvement worth the complexity it has added.
> This patch backs out all of the code specific to incremental
> processing, along with the persisting of OF flows,
> logical ports, multicast groups, all_lports, local and patched
> datapaths
>
> [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
>
> Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> Co-authored-by: Guru Shetty <g...@ovn.com>
>
> I'm mainly looking at binding.c right now, as that is code that I've
> gone through in detail since the incremental processing changes were
applied.
>
>
> diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
> index c26007d..0353a7b 100644
> --- a/ovn/controller/binding.c
> +++ b/ovn/controller/binding.c
> @@ -30,18 +30,6 @@
>
>  VLOG_DEFINE_THIS_MODULE(binding);
>
> -/* A set of the iface-id values of local interfaces on this chassis. */
> -static struct sset local_ids = SSET_INITIALIZER(_ids);
> -
> -/* When this gets set to true, the next run will re-check all
> binding records. */
> -static bool process_full_binding = false;
> -
> -void
> -binding_reset_processing(void)
> -{
> -    process_full_binding = true;
> -}
> -
>  void
>  binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
>  {
> @@ -64,16 +52,12 @@ binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
>                           _interface_col_ingress_policing_burst);
>  }
>
> -static bool
> +static void
>  get_local_iface_ids(const struct ovsrec_bridge *br_int,
>                      struct shash *lport_to_iface,
>                      struct sset *all_lports)
>  {
>      int i;
> -    bool changed = false;
> -
> -    struct sset old_local_ids = SSET_INITIALIZER(_local_ids);
> -    sset_clone(_local_ids, _ids);
>
>      for (i = 0; i < br_int->n_ports; i++) {
>          const struct ovsrec_port *port_rec = br_int->ports[i];
> @@ -93,53 +77,9 @@ get_local_iface_ids(const struct ovsrec_bridge
*br_int,
>                  continue;
>              }
>              shash_add(lport_to_iface, iface_id, iface_rec);
> -            if (!sset_find_and_delete(_local_ids, iface_id)) {
> -                sset_add(_ids, iface_id);
> -                sset_add(all_lports, iface_id);
> -                changed = true;
> -            }
> +            sset_add(all_lports, iface_id);
>          }
>      }
> -
> -    /* Any item left in old_local_ids is an ID for an interface
> -     * that has been removed. */
> -    if (!changed && !sset_is_empty(_local_ids)) {
> -        changed = true;
> -
> -        const char *cur_id;
> -        SSET_FOR_EACH(cur_id, _local_ids) {
> -            sset_find_and_delete(_ids, cur_id);
> -            sset_find_and_delete(all_lports, cur_id);
> -        }
> -    }
> -
> -    sset_destroy(_local_ids);
> -
> -    return changed;
> -}
> -
> -static struct local_datapath *
> -local_datapath_lookup_by_uuid(struct hmap *hmap_p, const struct uuid
*uuid)
> -{
> -    struct local_datapath *ld;
> -    HMAP_FOR_EACH_WITH_HASH(ld, uuid_hmap_node, uuid_hash(uuid), hmap_p)
{
> -        if (uuid_equals(>uuid, uuid)) {
> -            return ld;
> -        }
> -    }
> -    return NULL;
> -}
> -
> -static void
> -remove_local_datapath(struct hmap *local_datapaths, struct
> local_datapath *ld)
> -{
> -    if (ld->logical_port) {
> -        free(ld->logical_port);
> -        ld->logical_port = NULL;
> -    }
> -    hmap_remove(local_datapaths, >hmap_node);
> -    free(ld);
> -    lflow_reset_processing();
>  }

>
> This patch appears to break local_datapaths as it no longer ever
> removes anything from local_datapaths.
>
> all_lports also appears to have at least one problem.  localnet
> ports will never get removed from all_lports after this change.

Not true - as I said in the commit message, local_datapaths and
all_lports are no longer persisted, so they are rebuilt during
each cycle again...

>
>  static void
> @@ -156,9 +96,6 @@ add_local_datapath(struct hmap *local_datapaths,
>      memcpy(>uuid, _rec->header_.uuid, sizeof ld->uuid);
>      hmap_insert(local_datapaths, >hmap_node,
>                  binding_rec->datapath->tunnel

Re: [ovs-dev] [PATCH] ovn: fix ovn-northd leaks in build_acl

2016-08-25 Thread Ryan Moats 


"dev" <dev-boun...@openvswitch.org> wrote on 08/19/2016 08:06:50 PM:

> From: Ramu Ramamurthy <ramu.ramamur...@gmail.com>
> To: dev@openvswitch.org
> Cc: Suryanarayan Ramamurthy/San Jose/IBM@IBMUS
> Date: 08/19/2016 08:07 PM
> Subject: [ovs-dev] [PATCH] ovn: fix ovn-northd leaks in build_acl
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> From: Ramu Ramamurthy <ramu.ramamur...@us.ibm.com>
>
> The following leaks are due to missing ds_destroy in a few
> places in build_acl.
>
> 5,850 bytes in 50 blocks are definitely lost in loss record 93 of 93
>at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-
> amd64-linux.so)
>by 0x4C2BACB: realloc (in /usr/lib64/valgrind/vgpreload_memcheck-
> amd64-linux.so)
>by 0x449507: xrealloc (util.c:123)
>by 0x42CC73: ds_reserve (dynamic-string.c:63)
>by 0x42D08F: ds_put_format_valist (dynamic-string.c:161)
>by 0x42D176: ds_put_format (dynamic-string.c:142)
>by 0x40D380: build_acls (ovn-northd.c:2320)
>by 0x40D380: build_lswitch_flows.constprop.36 (ovn-northd.c:2472)
>by 0x4072D9: build_lflows (ovn-northd.c:3845)
>by 0x4072D9: ovnnb_db_run (ovn-northd.c:3971)
>by 0x4072D9: main (ovn-northd.c:4375)
>
> 9,360 bytes in 72 blocks are definitely lost in loss record 93 of 93
>at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-
> amd64-linux.so)
>by 0x4C2BACB: realloc (in /usr/lib64/valgrind/vgpreload_memcheck-
> amd64-linux.so)
>by 0x449507: xrealloc (util.c:123)
>by 0x42CC73: ds_reserve (dynamic-string.c:63)
>by 0x42D08F: ds_put_format_valist (dynamic-string.c:161)
>by 0x42D176: ds_put_format (dynamic-string.c:142)
>by 0x40D505: build_acls (ovn-northd.c:2346)
>by 0x40D505: build_lswitch_flows.constprop.36 (ovn-northd.c:2472)
>by 0x4072D9: build_lflows (ovn-northd.c:3845)
>by 0x4072D9: ovnnb_db_run (ovn-northd.c:3971)
>by 0x4072D9: main (ovn-northd.c:4375)
> ---
>  ovn/northd/ovn-northd.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/ovn/northd/ovn-northd.c b/ovn/northd/ovn-northd.c
> index 625937d..756d188 100644
> --- a/ovn/northd/ovn-northd.c
> +++ b/ovn/northd/ovn-northd.c
> @@ -2324,6 +2324,7 @@ build_acls(struct ovn_datapath *od, struct hmap
*lflows)
>  ovn_lflow_add(
>  lflows, od, S_SWITCH_OUT_ACL, 34000, ds_cstr
(),
>  actions);
> +ds_destroy();
>  }
>  }
>
> @@ -2350,6 +2351,7 @@ build_acls(struct ovn_datapath *od, struct hmap
*lflows)
>  ovn_lflow_add(
>  lflows, od, S_SWITCH_OUT_ACL, 34000, ds_cstr
(),
>  actions);
> +ds_destroy();
>  }
>  }
>  }
> --

Looks good to me on inspection, but I've not tried it out...

Acked-by: Ryan Moats <rmo...@us.ibm.com>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] ovn-northd: Avoid excessive work to find routerports.

2016-08-25 Thread Ryan Moats 


"dev" <dev-boun...@openvswitch.org> wrote on 08/17/2016 03:58:13 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: dev@openvswitch.org
> Cc: Ben Pfaff <b...@ovn.org>
> Date: 08/17/2016 03:58 PM
> Subject: [ovs-dev] [PATCH] ovn-northd: Avoid excessive work to find
> router ports.
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> The ovn_datapath for each logical switch maintains an array of its ports
> of type "router-port", but instead of iterating through it build_pre_acls
()
> iterated through all of the ports in the entire database, which is
> wasteful and duplicative work.  This commit switches to using the array
of
> router ports.
>
> This change is best viewed ignoring white space only changes.

LGTM

Acked-by: Ryan Moats <rmo...@us.ibm.com>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH 1/2] ovn/TODO: Update for items that have been completed.

2016-08-25 Thread Ryan Moats 
sis can
> @@ -144,38 +77,7 @@ large.
>needs work for scale and possibly for availability as deployments
>grow.  Here are some thoughts.
>
> -  Andy Zhou is looking at these issues.
> -
> -*** Reducing amount of data sent to clients.
> -
> -Currently, whenever a row monitored by a client changes,
> -ovsdb-server sends the client every monitored column in the row,
> -even if only one column changes.  It might be valuable to reduce
> -this only to the columns that changes.
> -
> -Also, whenever a column changes, ovsdb-server sends the entire
> -contents of the column.  It might be valuable, for columns that
> -are sets or maps, to send only added or removed values or
> -key-values pairs.
> -
> -Currently, clients monitor the entire contents of a table.  It
> -might make sense to allow clients to monitor only rows that
> -satisfy specific criteria, e.g. to allow an ovn-controller to
> -receive only Logical_Flow rows for logical networks on its
hypervisor.
> -
> -*** Reducing redundant data and code within ovsdb-server.
> -
> -Currently, ovsdb-server separately composes database update
> -information to send to each of its clients.  This is fine for a
> -small number of clients, but it wastes time and memory when
> -hundreds of clients all want the same updates (as will be in the
> -case in OVN).
> -
> -(This is somewhat opposed to the idea of letting a client monitor
> -only some rows in a table, since that would increase the diversity
> -among clients.)
> -
> -*** Multithreading.
> +** Multithreading.
>
>  If it turns out that other changes don't let ovsdb-server scale
>  adequately, we can multithread ovsdb-server.  Initially one might
> --

LGTM

Acked-by: Ryan Moats <rmo...@us.ibm.com>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] json: Use reference counting in json objects

2016-08-25 Thread Ryan Moats 


"dev"  wrote on 08/16/2016 04:50:37 PM:

> From: "Rodriguez Betancourt, Esteban" 
> To: "dev@openvswitch.org" 
> Date: 08/16/2016 04:50 PM
> Subject: [ovs-dev] [PATCH] json: Use reference counting in json objects
> Sent by: "dev" 
>
> After profiling OVSDB insert performance it was found
> that some significant portion of its time OVSDB is
> calling the function json_clone.
>
> Also, most of the usages of json_clone never modify the json,
> just keeps it to prevent it to be freed.
>
> With that in mind the struct json, json_create, json_clone
> and json_destroy were modified to keep a count of how many
> references of the json struct are left. Only when that count
> reaches zero the json struct is freed.
>
> The old "json_clone" function was renamed as "json_deep_clone".
>
> The change provides some performance improvement, depending
> on the transactions performed in OVSDB.
>
> Signed-off-by: Esteban Rodriguez Betancourt 
> ---

Can you add some quantification of the statement
"provides some performance improvement"?
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH v2] ovn: Delete stale MAC_Binding records

2016-08-25 Thread Ryan Moats 

"dev"  wrote on 08/16/2016 02:27:30 PM:

> From: Chandra Sekhar Vejendla/San Jose/IBM@IBMUS
> To: dev@openvswitch.org
> Date: 08/16/2016 02:33 PM
> Subject: [ovs-dev] [PATCH v2] ovn: Delete stale MAC_Binding records
> Sent by: "dev" 
>
> Entries in MAC_Binding table are not deleted when the logical_ports
> referred to in MAC_Bindings are deleted. The patch fixes this by
> deleting the MAC_Binding entry when the logical_port is not found.
>
> Signed-off-by: Chandra Sekhar Vejendla 
> ---
>  ovn/controller/lflow.c | 25 +
>  tests/ovn.at   | 25 +
>  2 files changed, 42 insertions(+), 8 deletions(-)
>
> diff --git a/ovn/controller/lflow.c b/ovn/controller/lflow.c
> index 341ca08..fbd36dc 100644
> --- a/ovn/controller/lflow.c
> +++ b/ovn/controller/lflow.c
> @@ -510,13 +510,11 @@ put_load(const uint8_t *data, size_t len,
>  }
>
>  static void
> -consider_neighbor_flow(const struct lport_index *lports,
> +consider_neighbor_flow(const struct sbrec_port_binding *pb,
> const struct sbrec_mac_binding *b,
> struct ofpbuf *ofpacts_p,
> struct match *match_p)
>  {
> -const struct sbrec_port_binding *pb
> -= lport_lookup_by_name(lports, b->logical_port);
>  if (!pb) {
>  return;
>  }
> @@ -571,10 +569,19 @@ add_neighbor_flows(struct controller_ctx *ctx,
>  match_init_catchall();
>  ofpbuf_init(, 0);
>
> -const struct sbrec_mac_binding *b;
> +const struct sbrec_port_binding *pb;
> +const struct sbrec_mac_binding *b, *n;
>  if (full_neighbor_flow_processing) {
> -SBREC_MAC_BINDING_FOR_EACH (b, ctx->ovnsb_idl) {
> -consider_neighbor_flow(lports, b, , );
> +SBREC_MAC_BINDING_FOR_EACH_SAFE (b, n, ctx->ovnsb_idl) {
> +/* Remove mac binding record if the port binding for the
logical
> + * port in mac binding record is not found. */
> +pb = lport_lookup_by_name(lports, b->logical_port);
> +if (!pb && ctx->ovnsb_idl_txn) {
> +ofctrl_remove_flows(>header_.uuid);
> +sbrec_mac_binding_delete(b);
> +continue;
> +}
> +consider_neighbor_flow(pb, b, , );
>  }
>  full_neighbor_flow_processing = false;
>  } else {
> @@ -585,11 +592,13 @@ add_neighbor_flows(struct controller_ctx *ctx,
>  if (!sbrec_mac_binding_is_new(b)) {
>  ofctrl_remove_flows(>header_.uuid);
>  }
> -consider_neighbor_flow(lports, b, , );
> +pb = lport_lookup_by_name(lports, b->logical_port);
> +if (pb) {
> +consider_neighbor_flow(pb, b, , );
> +}
>  }
>  }
>  }
> -
>  ofpbuf_uninit();
>  }
>
> diff --git a/tests/ovn.at b/tests/ovn.at
> index 216bb07..dbb205c 100644
> --- a/tests/ovn.at
> +++ b/tests/ovn.at
> @@ -4965,3 +4965,28 @@ cat packets
>  OVN_CLEANUP([hv1])
>
>  AT_CLEANUP
> +
> +AT_SETUP([ovn -- delete mac bindings])
> +AT_KEYWORDS([ovn])
> +ovn_start
> +net_add n1
> +sim_add hv1
> +as hv1
> +ovs-vsctl -- add-br br-phys
> +ovn_attach n1 br-phys 192.168.0.1
> +# Create logical switch ls0
> +ovn-nbctl ls-add ls0
> +# Create port lp0 in ls0
> +ovn-nbctl lsp-add ls0 lp0
> +ovn-nbctl lsp-set-addresses lp0 "f0:00:00:00:00:01 192.168.0.1"
> +dp_uuid=`ovn-sbctl find datapath | grep uuid | cut -f2 -d ":" | cut
> -f2 -d " "`
> +ovn-sbctl create MAC_Binding ip=10.0.0.1 datapath=$dp_uuid
> logical_port=lp0 mac="mac"
> +ovn-sbctl find MAC_Binding
> +#Delete port lp0
> +ovn-nbctl lsp-del lp0
> +ovn-sbctl find MAC_Binding
> +AT_CHECK([ovn-sbctl find MAC_Binding], [0], [])
> +
> +OVN_CLEANUP([hv1])
> +
> +AT_CLEANUP
> --

I think this should be abandoned in favor of
http://patchwork.ozlabs.org/patch/662152/ as trying to fix this
in ovn-controller rather than in ovn-northd will lead to race
conditions.
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH 2/3] daemon: Minor tweaking of man page fragment.

2016-08-25 Thread Ryan Moats 
Ryan Moats/Omaha/IBM wrote on 08/25/2016 09:19:49 AM:

> From: Ryan Moats/Omaha/IBM
> To: Justin Pettit <jpet...@ovn.org>
> Cc: dev@openvswitch.org
> Date: 08/25/2016 09:19 AM
> Subject: Re: [ovs-dev] [PATCH 2/3] daemon: Minor tweaking of man
> page fragment.
>
> "dev" <dev-boun...@openvswitch.org> wrote on 08/11/2016 01:09:13 AM:
>
> > From: Justin Pettit <jpet...@ovn.org>
> > To: dev@openvswitch.org
> > Date: 08/22/2016 01:36 PM
> > Subject: [ovs-dev] [PATCH 2/3] daemon: Minor tweaking of man page
fragment.
> > Sent by: "dev" <dev-boun...@openvswitch.org>
> >
> > Signed-off-by: Justin Pettit <jpet...@ovn.org>
> > ---
> >  lib/daemon.man | 2 +-
> >  lib/daemon.xml | 9 +
> >  2 files changed, 6 insertions(+), 5 deletions(-)
> >
> > diff --git a/lib/daemon.man b/lib/daemon.man
> > index f4e79ac..2855c2d 100644
> > --- a/lib/daemon.man
> > +++ b/lib/daemon.man
> > @@ -74,7 +74,7 @@ allowed, with current user or group are assumed
> > respectively. Only daemons
> >  started by the root user accepts this argument.
> >  .IP
> >  On Linux, daemons will be granted CAP_IPC_LOCK and
CAP_NET_BIND_SERVICES
> > -before dropping root privileges. Daemons interact with datapath,
> > +before dropping root privileges. Daemons that interact with a
datapath,
> >  such as ovs-vswitchd, will be granted two additional capabilities,
namely
> >  CAP_NET_ADMIN and CAP_NET_RAW. The capability change will apply even
if
> >  new user is "root".
> > diff --git a/lib/daemon.xml b/lib/daemon.xml
> > index d752e99..737ae55 100644
> > --- a/lib/daemon.xml
> > +++ b/lib/daemon.xml
> > @@ -106,10 +106,11 @@
> >  
> >On Linux, daemons will be granted CAP_IPC_LOCK and
> >CAP_NET_BIND_SERVICES before dropping root
privileges.
> > -  Daemons interact with datapath, such as ovs-
> vswitchd, will
> > -  be granted two additional capabilities, namely
> > CAP_NET_ADMIN
> > -  and CAP_NET_RAW.  The capability change will
> apply even if
> > -  the new user is root.
> > +  Daemons that interact with a datapath, such as
> > +  ovs-vswitchd, will be granted two additional
> > +  capabilities, namely CAP_NET_ADMIN and
> > +  CAP_NET_RAW.  The capability change will apply even
> > +  if the new user is root.
> >  
> >
> >  
> > --

> LGTM
>
> Acked-by: Ryan Moats <rmo...@us.ibm.com>

Well phooey - I thought I had the last in the series in my mailbox, but I
can't find it, so please consider the above an Ack for the whole series...

Ryan
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH 2/3] daemon: Minor tweaking of man page fragment.

2016-08-25 Thread Ryan Moats 


"dev" <dev-boun...@openvswitch.org> wrote on 08/11/2016 01:09:13 AM:

> From: Justin Pettit <jpet...@ovn.org>
> To: dev@openvswitch.org
> Date: 08/22/2016 01:36 PM
> Subject: [ovs-dev] [PATCH 2/3] daemon: Minor tweaking of man page
fragment.
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> Signed-off-by: Justin Pettit <jpet...@ovn.org>
> ---
>  lib/daemon.man | 2 +-
>  lib/daemon.xml | 9 +
>  2 files changed, 6 insertions(+), 5 deletions(-)
>
> diff --git a/lib/daemon.man b/lib/daemon.man
> index f4e79ac..2855c2d 100644
> --- a/lib/daemon.man
> +++ b/lib/daemon.man
> @@ -74,7 +74,7 @@ allowed, with current user or group are assumed
> respectively. Only daemons
>  started by the root user accepts this argument.
>  .IP
>  On Linux, daemons will be granted CAP_IPC_LOCK and CAP_NET_BIND_SERVICES
> -before dropping root privileges. Daemons interact with datapath,
> +before dropping root privileges. Daemons that interact with a datapath,
>  such as ovs-vswitchd, will be granted two additional capabilities,
namely
>  CAP_NET_ADMIN and CAP_NET_RAW. The capability change will apply even if
>  new user is "root".
> diff --git a/lib/daemon.xml b/lib/daemon.xml
> index d752e99..737ae55 100644
> --- a/lib/daemon.xml
> +++ b/lib/daemon.xml
> @@ -106,10 +106,11 @@
>  
>On Linux, daemons will be granted CAP_IPC_LOCK and
>CAP_NET_BIND_SERVICES before dropping root
privileges.
> -  Daemons interact with datapath, such as ovs-vswitchd,
will
> -  be granted two additional capabilities, namely
> CAP_NET_ADMIN
> -  and CAP_NET_RAW.  The capability change will apply
even if
> -  the new user is root.
> +  Daemons that interact with a datapath, such as
> +  ovs-vswitchd, will be granted two additional
> +  capabilities, namely CAP_NET_ADMIN and
> +  CAP_NET_RAW.  The capability change will apply even
> +  if the new user is root.
>  
>
>  
> --

LGTM

Acked-by: Ryan Moats <rmo...@us.ibm.com>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH 1/3] NEWS: Add reference to OVN in the post-2.5 release.

2016-08-25 Thread Ryan Moats 
"dev" <dev-boun...@openvswitch.org> wrote on 08/06/2016 10:51:24 PM:

> From: Justin Pettit <jpet...@ovn.org>
> To: dev@openvswitch.org
> Date: 08/15/2016 10:20 PM
> Subject: [ovs-dev] [PATCH 1/3] NEWS: Add reference to OVN in the
> post-2.5 release.
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> Signed-off-by: Justin Pettit <jpet...@ovn.org>
> ---
>  NEWS | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/NEWS b/NEWS
> index e989d64..53dc137 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -1,5 +1,7 @@
>  Post-v2.5.0
>  -
> +   - First supported release of OVN.  See ovn-architecture(7) for more
> + details.
> - ovsdb-server:
>   * New "monitor_cond" "monitor_cond_update" and "update2" extensions
to
> RFC 7047.
> --

Since patchsets 2 and 3 didn't make it to my mailbox, please consider this
an ack for the entire series that this patch starts...

Acked-by: Ryan Moats <rmo...@us.ibm.com>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v4] ovn-controller: Back out incremental processing

2016-08-24 Thread Ryan Moats 
As [1] indicates, incremental processing hasn't resulted
in an improvement worth the complexity it has added.
This patch backs out all of the code specific to incremental
processing, along with the persisting of OF flows,
logical ports, multicast groups, all_lports, local and patched
datapaths

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
Co-authored-by: Guru Shetty <g...@ovn.com>
---
 include/ovn/actions.h   |   4 -
 ovn/controller/binding.c| 154 ++-
 ovn/controller/binding.h|   1 -
 ovn/controller/encaps.c | 111 ++
 ovn/controller/lflow.c  | 102 -
 ovn/controller/lflow.h  |   4 +-
 ovn/controller/lport.c  | 220 +-
 ovn/controller/lport.h  |  24 +--
 ovn/controller/ofctrl.c | 331 +++-
 ovn/controller/ofctrl.h |  16 +-
 ovn/controller/ovn-controller.c |  60 +---
 ovn/controller/patch.c  |   6 -
 ovn/controller/physical.c   | 166 +---
 ovn/controller/physical.h   |   3 +-
 ovn/lib/actions.c   |   1 -
 tests/ovn.at|  55 +++
 16 files changed, 353 insertions(+), 905 deletions(-)

diff --git a/include/ovn/actions.h b/include/ovn/actions.h
index e2a716a..b678d33 100644
--- a/include/ovn/actions.h
+++ b/include/ovn/actions.h
@@ -298,7 +298,6 @@ struct group_table {
 struct group_info {
 struct hmap_node hmap_node;
 struct ds group;
-struct uuid lflow_uuid;
 uint32_t group_id;
 };
 
@@ -412,9 +411,6 @@ struct ovnact_encode_params {
 /* A struct to figure out the group_id for group actions. */
 struct group_table *group_table;
 
-/* The logical flow uuid that drove this action. */
-struct uuid lflow_uuid;
-
 /* OVN maps each logical flow table (ltable), one-to-one, onto a physical
  * OpenFlow flow table (ptable).  A number of parameters describe this
  * mapping and data related to flow tables:
diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
index c26007d..0353a7b 100644
--- a/ovn/controller/binding.c
+++ b/ovn/controller/binding.c
@@ -30,18 +30,6 @@
 
 VLOG_DEFINE_THIS_MODULE(binding);
 
-/* A set of the iface-id values of local interfaces on this chassis. */
-static struct sset local_ids = SSET_INITIALIZER(_ids);
-
-/* When this gets set to true, the next run will re-check all binding records. 
*/
-static bool process_full_binding = false;
-
-void
-binding_reset_processing(void)
-{
-process_full_binding = true;
-}
-
 void
 binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
 {
@@ -64,16 +52,12 @@ binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
  _interface_col_ingress_policing_burst);
 }
 
-static bool
+static void
 get_local_iface_ids(const struct ovsrec_bridge *br_int,
 struct shash *lport_to_iface,
 struct sset *all_lports)
 {
 int i;
-bool changed = false;
-
-struct sset old_local_ids = SSET_INITIALIZER(_local_ids);
-sset_clone(_local_ids, _ids);
 
 for (i = 0; i < br_int->n_ports; i++) {
 const struct ovsrec_port *port_rec = br_int->ports[i];
@@ -93,53 +77,9 @@ get_local_iface_ids(const struct ovsrec_bridge *br_int,
 continue;
 }
 shash_add(lport_to_iface, iface_id, iface_rec);
-if (!sset_find_and_delete(_local_ids, iface_id)) {
-sset_add(_ids, iface_id);
-sset_add(all_lports, iface_id);
-changed = true;
-}
+sset_add(all_lports, iface_id);
 }
 }
-
-/* Any item left in old_local_ids is an ID for an interface
- * that has been removed. */
-if (!changed && !sset_is_empty(_local_ids)) {
-changed = true;
-
-const char *cur_id;
-SSET_FOR_EACH(cur_id, _local_ids) {
-sset_find_and_delete(_ids, cur_id);
-sset_find_and_delete(all_lports, cur_id);
-}
-}
-
-sset_destroy(_local_ids);
-
-return changed;
-}
-
-static struct local_datapath *
-local_datapath_lookup_by_uuid(struct hmap *hmap_p, const struct uuid *uuid)
-{
-struct local_datapath *ld;
-HMAP_FOR_EACH_WITH_HASH(ld, uuid_hmap_node, uuid_hash(uuid), hmap_p) {
-if (uuid_equals(>uuid, uuid)) {
-return ld;
-}
-}
-return NULL;
-}
-
-static void
-remove_local_datapath(struct hmap *local_datapaths, struct local_datapath *ld)
-{
-if (ld->logical_port) {
-free(ld->logical_port);
-ld->logical_port = NULL;
-}
-hmap_remove(local_datapaths, >hmap_node);
-free(ld);
-lflow_reset_processing();
 }
 
 static void
@@ -156,9 +96,6 @@ add_local_datapath(struct hmap *local_datapaths,
 memcpy(>uuid, _rec->header_.uuid, sizeof ld->uuid);
 hmap_insert(loca

Re: [ovs-dev] [PATCH v3] ovn-controller: Back out incremental processing

2016-08-24 Thread Ryan Moats 



Guru Shetty <g...@ovn.org> wrote on 08/24/2016 03:57:41 PM:

> From: Guru Shetty <g...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: ovs dev <dev@openvswitch.org>
> Date: 08/24/2016 03:58 PM
> Subject: Re: [ovs-dev] [PATCH v3] ovn-controller: Back out
> incremental processing
>
> On 24 August 2016 at 11:12, Ryan Moats <rmo...@us.ibm.com> wrote:
>
>
>
> Guru Shetty <g...@ovn.org> wrote on 08/24/2016 01:07:47 PM:
>
> > From: Guru Shetty <g...@ovn.org>
> > To: Ryan Moats/Omaha/IBM@IBMUS
> > Cc: ovs dev <dev@openvswitch.org>
> > Date: 08/24/2016 01:08 PM
> > Subject: Re: [ovs-dev] [PATCH v3] ovn-controller: Back out
> > incremental processing
> >
> > On 24 August 2016 at 11:03, Ryan Moats <rmo...@us.ibm.com> wrote:
> >
> > Guru Shetty <g...@ovn.org> wrote on 08/24/2016 12:46:55 PM:
> >
> > > From: Guru Shetty <g...@ovn.org>
> > > To: Ryan Moats/Omaha/IBM@IBMUS
> > > Cc: ovs dev <dev@openvswitch.org>
> > > Date: 08/24/2016 12:47 PM
> > > Subject: Re: [ovs-dev] [PATCH v3] ovn-controller: Back out
> > > incremental processing
> > >
> > > On 23 August 2016 at 22:40, Ryan Moats <rmo...@us.ibm.com> wrote:
> > > As [1] indicates, incremental processing hasn't resulted
> > > in an improvement worth the complexity it has added.
> > > This patch backs out all of the code specific to incremental
> > > processing, along with the persisting of OF flows,
> > > logical ports and multicast groups.
> > >
> > > [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
> > >
> > > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> > >
> > > This is not a full review. But I have a few comments.
> > >
> > > * sparse gives the following warning
> > > ovn/controller/ofctrl.c:675:1: warning: symbol
> > > 'ovn_flow_table_clear' was not declared. Should it be static?
> >
> > hmm... I think I lost something because I remember needing to declare
> > that static...
> >
> > > * struct group_info still has lflow_uuid. Do we need it? It looks to
> > > me that it is not needed.
> >
> > I wanted to keep the change minimal to start with - I will look and
> > see if I can pull this out...
> >
> > > While you are at it, please replace the comment on top of ofctrl_put
> > > around groups to just read:
> > >      * Replaces the group table on the switch, if possible, by the
> > > 'groups->desired_groups'
> >
> > I can do that
> >
> > >
> > > * I notice that conntrack zone allocation for logical ports is still
> > > broken. I am not sure when it broke (but it has been broke for a
> > > long time), but I remember some patches for the fix around it for
> > > incremental processing
> > > For e.g., if you add the following test, you will notice that it
fails.
> >
> > With your permission, I'll add this in to PS 4 and get it fixed
> > You have my permission (I don't know what PS 4 is.). The test is
> > very minimalist and just tries to see whether atleast one logical
> > port has REG13 loaded as an example.

> Thanks, and PS 4 = Patch Set 4 (i.e. the next version)
>
> Just an FYI. If you fix the conntrack zone allocation correctly, 2
> load balancing tests will fail and that should just be because the
> output has an extra zone (which was previously just zero and I had
> missed the implication of it until yesterday.).

Well, I think I have, because I've undone the persistance of
all_lports, local_datapaths, and patched_datapaths in this go around...

I'm spinning v4 of the patch and emailing now, so we can collaborate
on how to fix the tests as part of v5 of the patch...

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH v3] ovn-controller: Back out incremental processing

2016-08-24 Thread Ryan Moats 



Guru Shetty <g...@ovn.org> wrote on 08/24/2016 01:07:47 PM:

> From: Guru Shetty <g...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: ovs dev <dev@openvswitch.org>
> Date: 08/24/2016 01:08 PM
> Subject: Re: [ovs-dev] [PATCH v3] ovn-controller: Back out
> incremental processing
>
> On 24 August 2016 at 11:03, Ryan Moats <rmo...@us.ibm.com> wrote:
>
> Guru Shetty <g...@ovn.org> wrote on 08/24/2016 12:46:55 PM:
>
> > From: Guru Shetty <g...@ovn.org>
> > To: Ryan Moats/Omaha/IBM@IBMUS
> > Cc: ovs dev <dev@openvswitch.org>
> > Date: 08/24/2016 12:47 PM
> > Subject: Re: [ovs-dev] [PATCH v3] ovn-controller: Back out
> > incremental processing
> >
> > On 23 August 2016 at 22:40, Ryan Moats <rmo...@us.ibm.com> wrote:
> > As [1] indicates, incremental processing hasn't resulted
> > in an improvement worth the complexity it has added.
> > This patch backs out all of the code specific to incremental
> > processing, along with the persisting of OF flows,
> > logical ports and multicast groups.
> >
> > [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
> >
> > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> >
> > This is not a full review. But I have a few comments.
> >
> > * sparse gives the following warning
> > ovn/controller/ofctrl.c:675:1: warning: symbol
> > 'ovn_flow_table_clear' was not declared. Should it be static?
>
> hmm... I think I lost something because I remember needing to declare
> that static...
>
> > * struct group_info still has lflow_uuid. Do we need it? It looks to
> > me that it is not needed.
>
> I wanted to keep the change minimal to start with - I will look and
> see if I can pull this out...
>
> > While you are at it, please replace the comment on top of ofctrl_put
> > around groups to just read:
> >      * Replaces the group table on the switch, if possible, by the
> > 'groups->desired_groups'
>
> I can do that
>
> >
> > * I notice that conntrack zone allocation for logical ports is still
> > broken. I am not sure when it broke (but it has been broke for a
> > long time), but I remember some patches for the fix around it for
> > incremental processing
> > For e.g., if you add the following test, you will notice that it fails.
>
> With your permission, I'll add this in to PS 4 and get it fixed
> You have my permission (I don't know what PS 4 is.). The test is
> very minimalist and just tries to see whether atleast one logical
> port has REG13 loaded as an example.

Thanks, and PS 4 = Patch Set 4 (i.e. the next version)


>
> >
> > AT_SETUP([ovn -- conntrack zone allocation])
> > AT_KEYWORDS([ovnconntrack])
> > AT_SKIP_IF([test $HAVE_PYTHON = no])
> > ovn_start
> >
> > # Logical network:
> > # 2 logical switches "foo" (192.168.1.0/24) and "bar" (172.16.1.0/24)
> > # connected to a router R1.
> > # foo has foo1 to act as a client.
> > # bar has bar1, bar2, bar3 to act as servers.
> >
> > net_add n1
> >
> > sim_add hv1
> > as hv1
> > ovs-vsctl add-br br-phys
> > ovn_attach n1 br-phys 192.168.0.1
> > for i in foo1 bar1 bar2 bar3; do
> >     ovs-vsctl -- add-port br-int $i -- \
> >         set interface $i external-ids:iface-id=$i \
> >         options:tx_pcap=hv1/$i-tx.pcap \
> >         options:rxq_pcap=hv1/$i-rx.pcap
> > done
> >
> > ovn-nbctl create Logical_Router name=R1
> > ovn-nbctl ls-add foo
> > ovn-nbctl ls-add bar
> >
> > # Connect foo to R1
> > ovn-nbctl lrp-add R1 foo 00:00:01:01:02:03 192.168.1.1/24
> > ovn-nbctl lsp-add foo rp-foo -- set Logical_Switch_Port rp-foo \
> >     type=router options:router-port=foo addresses=\"00:00:01:01:02:03\"
> >
> > # Connect bar to R1
> > ovn-nbctl lrp-add R1 bar 00:00:01:01:02:04 172.16.1.1/24
> > ovn-nbctl lsp-add bar rp-bar -- set Logical_Switch_Port rp-bar \
> >     type=router options:router-port=bar addresses=\"00:00:01:01:02:04\"
> >
> > # Create logical port foo1 in foo
> > ovn-nbctl lsp-add foo foo1 \
> > -- lsp-set-addresses foo1 "f0:00:00:01:02:03 192.168.1.2"
> >
> > # Create logical port bar1, bar2 and bar3 in bar
> > for i in `seq 1 3`; do
> >     ip=`expr $i + 1`
> >     ovn-nbctl lsp-add bar bar$i \
> >     -- lsp-set-addresses bar$i "f0:00:0a:01:02:$i 172.16.1.$ip"
> > done
> >
> > OVS_WAIT_UNTIL([ovs-ofctl dump-flows br-int table=0 | grep REG13])
> >
> > OVN_CLEANUP([hv1])
> >
> > AT_CL

Re: [ovs-dev] [PATCH v3] ovn-controller: Back out incremental processing

2016-08-24 Thread Ryan Moats 

Guru Shetty <g...@ovn.org> wrote on 08/24/2016 12:46:55 PM:

> From: Guru Shetty <g...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: ovs dev <dev@openvswitch.org>
> Date: 08/24/2016 12:47 PM
> Subject: Re: [ovs-dev] [PATCH v3] ovn-controller: Back out
> incremental processing
>
> On 23 August 2016 at 22:40, Ryan Moats <rmo...@us.ibm.com> wrote:
> As [1] indicates, incremental processing hasn't resulted
> in an improvement worth the complexity it has added.
> This patch backs out all of the code specific to incremental
> processing, along with the persisting of OF flows,
> logical ports and multicast groups.
>
> [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
>
> Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
>
> This is not a full review. But I have a few comments.
>
> * sparse gives the following warning
> ovn/controller/ofctrl.c:675:1: warning: symbol
> 'ovn_flow_table_clear' was not declared. Should it be static?

hmm... I think I lost something because I remember needing to declare
that static...

> * struct group_info still has lflow_uuid. Do we need it? It looks to
> me that it is not needed.

I wanted to keep the change minimal to start with - I will look and
see if I can pull this out...

> While you are at it, please replace the comment on top of ofctrl_put
> around groups to just read:
>      * Replaces the group table on the switch, if possible, by the
> 'groups->desired_groups'

I can do that

>
> * I notice that conntrack zone allocation for logical ports is still
> broken. I am not sure when it broke (but it has been broke for a
> long time), but I remember some patches for the fix around it for
> incremental processing
> For e.g., if you add the following test, you will notice that it fails.

With your permission, I'll add this in to PS 4 and get it fixed

>
> AT_SETUP([ovn -- conntrack zone allocation])
> AT_KEYWORDS([ovnconntrack])
> AT_SKIP_IF([test $HAVE_PYTHON = no])
> ovn_start
>
> # Logical network:
> # 2 logical switches "foo" (192.168.1.0/24) and "bar" (172.16.1.0/24)
> # connected to a router R1.
> # foo has foo1 to act as a client.
> # bar has bar1, bar2, bar3 to act as servers.
>
> net_add n1
>
> sim_add hv1
> as hv1
> ovs-vsctl add-br br-phys
> ovn_attach n1 br-phys 192.168.0.1
> for i in foo1 bar1 bar2 bar3; do
>     ovs-vsctl -- add-port br-int $i -- \
>         set interface $i external-ids:iface-id=$i \
>         options:tx_pcap=hv1/$i-tx.pcap \
>         options:rxq_pcap=hv1/$i-rx.pcap
> done
>
> ovn-nbctl create Logical_Router name=R1
> ovn-nbctl ls-add foo
> ovn-nbctl ls-add bar
>
> # Connect foo to R1
> ovn-nbctl lrp-add R1 foo 00:00:01:01:02:03 192.168.1.1/24
> ovn-nbctl lsp-add foo rp-foo -- set Logical_Switch_Port rp-foo \
>     type=router options:router-port=foo addresses=\"00:00:01:01:02:03\"
>
> # Connect bar to R1
> ovn-nbctl lrp-add R1 bar 00:00:01:01:02:04 172.16.1.1/24
> ovn-nbctl lsp-add bar rp-bar -- set Logical_Switch_Port rp-bar \
>     type=router options:router-port=bar addresses=\"00:00:01:01:02:04\"
>
> # Create logical port foo1 in foo
> ovn-nbctl lsp-add foo foo1 \
> -- lsp-set-addresses foo1 "f0:00:00:01:02:03 192.168.1.2"
>
> # Create logical port bar1, bar2 and bar3 in bar
> for i in `seq 1 3`; do
>     ip=`expr $i + 1`
>     ovn-nbctl lsp-add bar bar$i \
>     -- lsp-set-addresses bar$i "f0:00:0a:01:02:$i 172.16.1.$ip"
> done
>
> OVS_WAIT_UNTIL([ovs-ofctl dump-flows br-int table=0 | grep REG13])
>
> OVN_CLEANUP([hv1])
>
> AT_CLEANUP
>
>
>
>
> ---
>  v1->v2:
>  - removed some obvious memory leaks left behind in physical.c
>  v2->v3:
>  - removed some less obvious memory leaks created by
>    non-persisting ofctrl.c
>
>  ovn/controller/binding.c        | 120 +--
>  ovn/controller/binding.h        |   1 -
>  ovn/controller/encaps.c         | 111 ++
>  ovn/controller/lflow.c          | 101 -
>  ovn/controller/lflow.h          |   4 +-
>  ovn/controller/lport.c          | 220 +--
>  ovn/controller/lport.h          |  24 +--
>  ovn/controller/ofctrl.c         | 323 ++
> +-
>  ovn/controller/ofctrl.h         |  16 +-
>  ovn/controller/ovn-controller.c |  26 ++--
>  ovn/controller/patch.c          |   6 -
>  ovn/controller/physical.c       | 166 +
>  ovn/controller/physical.h       |   3 +-
>  13 files changed, 295 insertions(+), 826 deletions(-)

>
> diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
> index c26007d..f2552fa 100644
> --- a/ovn/co

[ovs-dev] [PATCH] ovn-controller: Fix memory leak when parsing lflow actions.

2016-08-24 Thread Ryan Moats 
Parsing logical flow actions with ovnacts_parse* that include
string constants current leak memory.  Add calls to ovnacts_free
to recapture said memory.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/lflow.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/ovn/controller/lflow.c b/ovn/controller/lflow.c
index 341ca08..40aa49d 100644
--- a/ovn/controller/lflow.c
+++ b/ovn/controller/lflow.c
@@ -404,6 +404,7 @@ consider_logical_flow(const struct lport_index *lports,
 VLOG_WARN_RL(, "error parsing actions \"%s\": %s",
  lflow->actions, error);
 free(error);
+ovnacts_free(ovnacts.data, ovnacts.size);
 ofpbuf_uninit();
 return;
 }
@@ -428,6 +429,7 @@ consider_logical_flow(const struct lport_index *lports,
 .mac_bind_ptable = OFTABLE_MAC_BINDING,
 };
 ovnacts_encode(ovnacts.data, ovnacts.size, , );
+ovnacts_free(ovnacts.data, ovnacts.size);
 ofpbuf_uninit();
 
 /* Translate OVN match into table of OpenFlow matches. */
-- 
2.7.4

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] ovn: Delete stale MAC_Bindings that result in Referential Integrity Violation

2016-08-24 Thread Ryan Moats 
"dev" <dev-boun...@openvswitch.org> wrote on 08/24/2016 02:10:29 AM:

> From: Chandra Sekhar Vejendla/San Jose/IBM@IBMUS
> To: dev@openvswitch.org
> Date: 08/24/2016 02:11 AM
> Subject: [ovs-dev] [PATCH] ovn: Delete stale MAC_Bindings that
> result in Referential Integrity Violation
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> The MAC_Bindings have a strong reference to the Datapath_Binding. However
the
> MAC_Bindings are never deleted anywhere, and when the Datapath
(associated
> with a MAC_Binding) is deleted, the ovsdb-server returns Referential
> Integrity Violation. This prevents newer operations initiated from the
CMS
> from being committed to the Southbound DB.
>
> The patch fixes this  by deleting the MAC_Binding entry when the
> logical_port referred in the mac_binding entry is deleted.
>
> Signed-off-by: Chandra Sekhar Vejendla <csvej...@us.ibm.com>
> ---

I've run the test without and with the ovn-northd.c patch and verified
that without the patch it fails, and with the patch it succeeds.

I've followed this up by stacking the patch with tip of the tree
openstack master and verified that without the patch, trying to clean
up the default router made by stacking does not clean up the
Datapath_Binding because of a stale MAC_Binding entry, while with the
patch, everything cleans up correctly.

Lastly, I've run the stock rally create-and-delete-routers job and verified
that all Datapath_Binding entries for routers created by rally are cleaned
up correctly (i.e. there are no stale MAC_Binding entries), so...

Acked-by: Ryan Moats <rmo...@us.ibm.com>
Tested-by: Ryan Moats <rmo...@us.ibm.com>

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v3] ovn-controller: Back out incremental processing

2016-08-23 Thread Ryan Moats 
As [1] indicates, incremental processing hasn't resulted
in an improvement worth the complexity it has added.
This patch backs out all of the code specific to incremental
processing, along with the persisting of OF flows,
logical ports and multicast groups.

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 v1->v2:
 - removed some obvious memory leaks left behind in physical.c
 v2->v3:
 - removed some less obvious memory leaks created by
   non-persisting ofctrl.c 

 ovn/controller/binding.c| 120 +--
 ovn/controller/binding.h|   1 -
 ovn/controller/encaps.c | 111 ++
 ovn/controller/lflow.c  | 101 -
 ovn/controller/lflow.h  |   4 +-
 ovn/controller/lport.c  | 220 +--
 ovn/controller/lport.h  |  24 +--
 ovn/controller/ofctrl.c | 323 +++-
 ovn/controller/ofctrl.h |  16 +-
 ovn/controller/ovn-controller.c |  26 ++--
 ovn/controller/patch.c  |   6 -
 ovn/controller/physical.c   | 166 +
 ovn/controller/physical.h   |   3 +-
 13 files changed, 295 insertions(+), 826 deletions(-)

diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
index c26007d..f2552fa 100644
--- a/ovn/controller/binding.c
+++ b/ovn/controller/binding.c
@@ -33,15 +33,6 @@ VLOG_DEFINE_THIS_MODULE(binding);
 /* A set of the iface-id values of local interfaces on this chassis. */
 static struct sset local_ids = SSET_INITIALIZER(_ids);
 
-/* When this gets set to true, the next run will re-check all binding records. 
*/
-static bool process_full_binding = false;
-
-void
-binding_reset_processing(void)
-{
-process_full_binding = true;
-}
-
 void
 binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
 {
@@ -139,7 +130,6 @@ remove_local_datapath(struct hmap *local_datapaths, struct 
local_datapath *ld)
 }
 hmap_remove(local_datapaths, >hmap_node);
 free(ld);
-lflow_reset_processing();
 }
 
 static void
@@ -156,9 +146,6 @@ add_local_datapath(struct hmap *local_datapaths,
 memcpy(>uuid, _rec->header_.uuid, sizeof ld->uuid);
 hmap_insert(local_datapaths, >hmap_node,
 binding_rec->datapath->tunnel_key);
-lport_index_reset();
-mcgroup_index_reset();
-lflow_reset_processing();
 }
 
 static void
@@ -268,80 +255,49 @@ binding_run(struct controller_ctx *ctx, const struct 
ovsrec_bridge *br_int,
 }
 
 if (br_int) {
-if (ctx->ovnsb_idl_txn && get_local_iface_ids(br_int, _to_iface,
-  all_lports)) {
-process_full_binding = true;
-}
-} else {
-/* We have no integration bridge, therefore no local logical ports.
- * We'll remove our chassis from all port binding records below. */
-process_full_binding = true;
+get_local_iface_ids(br_int, _to_iface, all_lports);
 }
 
-/* Run through each binding record to see if it is resident on this
- * chassis and update the binding accordingly.  This includes both
- * directly connected logical ports and children of those ports. */
-if (process_full_binding) {
-/* Detect any entries in all_lports that have been deleted.
- * In particular, this will catch localnet ports that we
- * put in all_lports. */
-struct sset removed_lports = SSET_INITIALIZER(_lports);
-sset_clone(_lports, all_lports);
-
-struct hmap keep_local_datapath_by_uuid =
-HMAP_INITIALIZER(_local_datapath_by_uuid);
-SBREC_PORT_BINDING_FOR_EACH(binding_rec, ctx->ovnsb_idl) {
-sset_find_and_delete(_lports, binding_rec->logical_port);
-consider_local_datapath(ctx, chassis_rec, binding_rec,
-local_datapaths, _to_iface,
-all_lports);
-struct local_datapath *ld = xzalloc(sizeof *ld);
-memcpy(>uuid, _rec->header_.uuid, sizeof ld->uuid);
-hmap_insert(_local_datapath_by_uuid, >uuid_hmap_node,
-uuid_hash(>uuid));
-}
-struct local_datapath *old_ld, *next;
-HMAP_FOR_EACH_SAFE (old_ld, next, hmap_node, local_datapaths) {
-if (!local_datapath_lookup_by_uuid(_local_datapath_by_uuid,
-   _ld->uuid)) {
-remove_local_datapath(local_datapaths, old_ld);
-}
-}
-struct local_datapath *ld;
-HMAP_FOR_EACH_SAFE (ld, next, uuid_hmap_node,
-_local_datapath_by_uuid) {
-hmap_remove(_local_datapath_by_uuid, >uuid_hmap_node);
-free(ld);
-}
-hmap_destroy(_local_datapath_by_uuid);
+/* Detect any entries in all_lports that have been

[ovs-dev] [PATCH v2] ovn-controller: Back out incremental processing

2016-08-23 Thread Ryan Moats 
As [1] indicates, incremental processing hasn't resulted
in an improvement worth the complexity it has added.
This patch backs out all of the code specific to incremental
processing, along with the persisting of OF flows,
logical ports and multicast groups.

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 v1->v2:
   removed some obvious memory leaks left behind in physical.c

 ovn/controller/binding.c| 120 +---
 ovn/controller/binding.h|   1 -
 ovn/controller/encaps.c | 111 ++-
 ovn/controller/lflow.c  | 101 -
 ovn/controller/lflow.h  |   4 +-
 ovn/controller/lport.c  | 220 +
 ovn/controller/lport.h  |  24 +---
 ovn/controller/ofctrl.c | 306 +---
 ovn/controller/ofctrl.h |  16 +--
 ovn/controller/ovn-controller.c |  26 ++--
 ovn/controller/patch.c  |   6 -
 ovn/controller/physical.c   | 166 ++
 ovn/controller/physical.h   |   3 +-
 13 files changed, 279 insertions(+), 825 deletions(-)

diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
index c26007d..f2552fa 100644
--- a/ovn/controller/binding.c
+++ b/ovn/controller/binding.c
@@ -33,15 +33,6 @@ VLOG_DEFINE_THIS_MODULE(binding);
 /* A set of the iface-id values of local interfaces on this chassis. */
 static struct sset local_ids = SSET_INITIALIZER(_ids);
 
-/* When this gets set to true, the next run will re-check all binding records. 
*/
-static bool process_full_binding = false;
-
-void
-binding_reset_processing(void)
-{
-process_full_binding = true;
-}
-
 void
 binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
 {
@@ -139,7 +130,6 @@ remove_local_datapath(struct hmap *local_datapaths, struct 
local_datapath *ld)
 }
 hmap_remove(local_datapaths, >hmap_node);
 free(ld);
-lflow_reset_processing();
 }
 
 static void
@@ -156,9 +146,6 @@ add_local_datapath(struct hmap *local_datapaths,
 memcpy(>uuid, _rec->header_.uuid, sizeof ld->uuid);
 hmap_insert(local_datapaths, >hmap_node,
 binding_rec->datapath->tunnel_key);
-lport_index_reset();
-mcgroup_index_reset();
-lflow_reset_processing();
 }
 
 static void
@@ -268,80 +255,49 @@ binding_run(struct controller_ctx *ctx, const struct 
ovsrec_bridge *br_int,
 }
 
 if (br_int) {
-if (ctx->ovnsb_idl_txn && get_local_iface_ids(br_int, _to_iface,
-  all_lports)) {
-process_full_binding = true;
-}
-} else {
-/* We have no integration bridge, therefore no local logical ports.
- * We'll remove our chassis from all port binding records below. */
-process_full_binding = true;
+get_local_iface_ids(br_int, _to_iface, all_lports);
 }
 
-/* Run through each binding record to see if it is resident on this
- * chassis and update the binding accordingly.  This includes both
- * directly connected logical ports and children of those ports. */
-if (process_full_binding) {
-/* Detect any entries in all_lports that have been deleted.
- * In particular, this will catch localnet ports that we
- * put in all_lports. */
-struct sset removed_lports = SSET_INITIALIZER(_lports);
-sset_clone(_lports, all_lports);
-
-struct hmap keep_local_datapath_by_uuid =
-HMAP_INITIALIZER(_local_datapath_by_uuid);
-SBREC_PORT_BINDING_FOR_EACH(binding_rec, ctx->ovnsb_idl) {
-sset_find_and_delete(_lports, binding_rec->logical_port);
-consider_local_datapath(ctx, chassis_rec, binding_rec,
-local_datapaths, _to_iface,
-all_lports);
-struct local_datapath *ld = xzalloc(sizeof *ld);
-memcpy(>uuid, _rec->header_.uuid, sizeof ld->uuid);
-hmap_insert(_local_datapath_by_uuid, >uuid_hmap_node,
-uuid_hash(>uuid));
-}
-struct local_datapath *old_ld, *next;
-HMAP_FOR_EACH_SAFE (old_ld, next, hmap_node, local_datapaths) {
-if (!local_datapath_lookup_by_uuid(_local_datapath_by_uuid,
-   _ld->uuid)) {
-remove_local_datapath(local_datapaths, old_ld);
-}
-}
-struct local_datapath *ld;
-HMAP_FOR_EACH_SAFE (ld, next, uuid_hmap_node,
-_local_datapath_by_uuid) {
-hmap_remove(_local_datapath_by_uuid, >uuid_hmap_node);
-free(ld);
-}
-hmap_destroy(_local_datapath_by_uuid);
+/* Detect any entries in all_lports that have been deleted.
+ * In particular, this will catch localnet ports that we
+ * put

Re: [ovs-dev] [PATCH v1] ovn: ensure datapath removal is proper

2016-08-23 Thread Ryan Moats 
"dev"  wrote on 08/22/2016 08:56:50 PM:

> From: Flavio Fernandes 
> To: dev@openvswitch.org
> Date: 08/22/2016 08:57 PM
> Subject: [ovs-dev] [PATCH v1] ovn: ensure datapath removal is proper
> Sent by: "dev" 
>
> Adding a unit test in ovn.at, to exercise the cleanup of
> OF rules related to a logical datapath, when a logical
> switch is removed.
>
> Reported-by: Guru Shetty 
> Reported-at:
http://openvswitch.org/pipermail/discuss/2016-August/022478.html
> Signed-off-by: Flavio Fernandes 
> ---
> NOTE: This test is expected to fail at this time, as the bug is not yet
fixed.
>
>  tests/ovn.at | 58 ++

>  1 file changed, 58 insertions(+)
>
> diff --git a/tests/ovn.at b/tests/ovn.at
> index 216bb07..604cdcb 100644
> --- a/tests/ovn.at
> +++ b/tests/ovn.at
> @@ -4319,6 +4319,64 @@ OVN_CLEANUP([hv1])
>
>  AT_CLEANUP
>
> +# 1 hypervisor, 1 port
> +# make sure that the OF rules created to support a datapath are
added/cleared
> +# when logical switch is created and removed.
> +AT_SETUP([ovn -- datapath rules added/removed])
> +AT_KEYWORDS([ovn datapath cleanup])
> +ovn_start
> +
> +net_add n1
> +sim_add hv1
> +as hv1 ovs-vsctl add-br br-phys
> +as hv1 ovn_attach n1 br-phys 192.168.0.1
> +
> +# This shell function checks if OF rules in br-int have clauses
> +# related to OVN datapaths. The caller determines if it should find
> +# a match in the output, or not.
> +test_datapath_in_of_rules() {
> +local expect_datapath=$1 stage_info=$2
> +echo "-- ovn-nbctl show ${stage_info} --"
> +ovn-nbctl show
> +echo "-- ovn-sbctl show ${stage_info} --"
> +ovn-sbctl show
> +echo "-- OF rules ${stage_info} --"
> +AT_CHECK([ovs-ofctl dump-flows br-int], [0], [stdout])
> +# if there is a datapath mentioned in the output, check for the
> +# magic keyword that represents one, based on the exit status of
> +# a quiet grep
> +if test $expect_datapath != 0; then
> +   AT_CHECK([grep --quiet -i 'metadata=' stdout], [0],
[ignore-nolog])
> +else
> +   AT_CHECK([grep --quiet -i 'metadata=' stdout], [1],
[ignore-nolog])
> +fi
> +}
> +
> +test_datapath_in_of_rules 0 "before ls+port create"
> +
> +ovn-nbctl ls-add ls1
> +ovn-nbctl lsp-add ls1 lp1
> +ovn-nbctl lsp-set-addresses lp1 unknown
> +
> +as hv1 ovs-vsctl add-port br-int vif1 -- set Interface vif1
> external-ids:iface-id=lp1
> +OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up lp1` = xup])
> +
> +test_datapath_in_of_rules 1 "after port is bound"
> +
> +as hv1 ovs-vsctl del-port br-int vif1
> +OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up lp1` = xdown])
> +
> +ovn-nbctl lsp-set-addresses lp1
> +ovn-nbctl lsp-del lp1
> +ovn-nbctl ls-del ls1
> +
> +# NOTE: if a known bug is present, THIS WILL BREAK HERE
> +test_datapath_in_of_rules 0 "after ls+port removal"
> +
> +OVN_CLEANUP([hv1])
> +
> +AT_CLEANUP
> +
>  AT_SETUP([ovn -- nd_na ])
>  AT_KEYWORDS([ovn-nd_na])
>  AT_SKIP_IF([test $HAVE_PYTHON = no])
> --

I agree that this patch doesn't work when applied to current
master.  When I tried it against the patch that backs out incremental
processing "ovn-controller: Back out incremental processing"
then it works, so I'm not that inclined to chase an independent
solution to it that hard.

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [RFC] ovn: minimize the impact of a compromised chassis

2016-08-22 Thread Ryan Moats 


Russell Bryant <rbry...@redhat.com> wrote on 08/22/2016 07:56:41 AM:

> From: Russell Bryant <rbry...@redhat.com>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: Justin Pettit <jpet...@ovn.org>, ovs dev <dev@openvswitch.org>
> Date: 08/22/2016 07:57 AM
> Subject: Re: [ovs-dev] [RFC] ovn: minimize the impact of a
compromisedchassis
>
> On Sun, Aug 21, 2016 at 9:06 PM, Ryan Moats <rmo...@us.ibm.com> wrote:
> "dev" <dev-boun...@openvswitch.org> wrote on 08/21/2016 03:02:12 PM:
>
> > From: Justin Pettit <jpet...@ovn.org>
> > To: Russell Bryant <rbry...@redhat.com>
> > Cc: ovs dev <dev@openvswitch.org>
> > Date: 08/21/2016 07:30 PM
> > Subject: Re: [ovs-dev] [RFC] ovn: minimize the impact of a
> compromised chassis
> > Sent by: "dev" <dev-boun...@openvswitch.org>
>
> >
> >
> > > On Aug 16, 2016, at 6:52 AM, Russell Bryant <rbry...@redhat.com>
wrote:
> > >
> > > ​Thanks for starting this discussion.​
> > >
> > > ​I do think making ovn-controller a read-only client of the database
seems
> > > like the simplest path forward.  We should pursue that until we hit a
> > > strong reason not to.
> > >
> > > One of the biggest questions remaining for me is how we deal
> with backwards
> > > compatibility.  Whatever we do here will have to account for what
happens
> > > for environments running OVN from OVS 2.6 when they upgrade.
> > >
> > > Perhaps the most straight forward way to do that is to make this new
more
> > > secure mode optional and off by default.  The downside is that
> it makes the
> > > system more complex, since it will have different modes it can work.
> >
> > I think we should avoid providing two modes if at all possible.  As
> > you mentioned, it increases the complexity, which will likely make
> > it more difficult to test thoroughly and deploy consistently.
> >
> > > An alternative would be to provide some tooling to assist with
> the upgrade:
> > >
> > > - Document the new requirements for creating Chassis and Encap rows
> > > manually
> > >
> > > - Provide an upgrade tool (via ovn-nbctl/ovn-sbctl/something-else)
that
> > > will add the "chassis" option to logical ports set based on
> where ports are
> > > currently bound in the southbound database.
> > >
> > > ​ - Provide an upgrade tool that converts the MAC_Binding table
> to whatever
> > > the new chassis local storage for that information would be.
> >
> > I think the tool approach is better.  I imagine people deploying 2.6
> > will be pretty closely tied to the community still, so walking them
> > through an upgrade shouldn't be too bad--especially if we make it
> > easy.  Obviously, we'll try to avoid the need for that again in the
future.
> >
> > --Justin

> I agree that the ovn-controller process should be limited to read-only
> for the following tables
>      SB_Global
>      Logical_Flow
>      Multicast_Group
>      Datapath_Binding
>      Address_Set
>      DHCP_Options
>      DHCPv6_Options
>
> However, I'm going to argue that the suggestions for making
ovn-controller
> read-only for
>
>  Chassis
>  Encap
>  Port_Binding
>  MAC_Binding
>
> need some more discussion as I don't think all of the suggestions to date
> are entirely feasible...
>
>
>
> First, putting the responsibility for updating the Chassis and Encap
tables
> on the CMS require (a) that the CMS have the ability to provide that
> information
> and (b) that the CMS have an OVN plugin.  I see this as moving OVN out of
the
> pure networking space (at least with respect to OpenStack) as I don't see
> Neutron getting this functionality any time soon.  I agree with Justin
that
> the right approach is to provide documentation for how to
add/remove/update
> entries via ovn-sbctl and then leave it to operators to incorporate that
> tooling into their deployment scenarios.
>

> ​I never suggested that the CMS do this.  :-)
>
> I think it should be an administrative task that is part of the
> deployment process.​

We are in agreement then...

>
> For Port_Binding, while it is possible to get that information from the
CMS,
> I worry (at least in the OpenStack case) about a potential race condition
> during instance boot - we've already seen cases during scale testingwhere
the
> current method (having the chassis update the port binding) doesn't
percolate
> through Neutron back to Nova fast enough, and using the CMS to set the
> port binding will add another half round trip

[ovs-dev] [CudaMailTagged] Re: [PATCH] ovn-controller: add quiet mode

2016-08-22 Thread Ryan Moats 
Hui Kang/Watson/IBM wrote on 08/21/2016 10:23:21 PM:

> From: Hui Kang/Watson/IBM
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/21/2016 10:23 PM
> Subject: Re: [ovs-dev] [PATCH] ovn-controller: add quiet mode
>
> I did observe that the previous incremental patch improves the ova-
> controller's CPU utilization a lot.
>
> I will test these two patch to ensure that the improvement will not
> disappear. Thanks.
>
> - Hui

Hui-

Please also consider v2 of the "add quiet mode" patch for this. I
put v1 in a four node OpenStack cloud and found that
ovn-controller didn't quiesce once I had a patch port on br-int.
v2 addresses this issue, but I believe there is still an issue once
I have a logical router connecting two logical switches.  While I'm
looking at this myself today, an independent set of eyes would be
appreciated.

Ryan
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v2] ovn-controller: add quiet mode

2016-08-21 Thread Ryan Moats 
As discussed in [1], what the incremental processing code
actually accomplished was that the ovn-controller would
be "quiet" and not burn CPU when things weren't changing.
This patch set recreates this state by calculating whether
changes have occured that would require a full calculation
to be performed.

This commit depends on f5d916cb ("ovn-controller:
Back out incremental processing")

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 v2->v1:
   - moved calls to force_full_process in patch.c to avoid defeating
 quiet mode when dealing with patch ports between bridges
   - added TODO comment to patch.c for Logical Router case
 ovn/controller/ofctrl.c |  2 ++
 ovn/controller/ovn-controller.c | 77 +
 ovn/controller/ovn-controller.h |  1 +
 ovn/controller/patch.c  | 11 ++
 ovn/controller/physical.c   | 55 +++--
 ovn/controller/physical.h   |  5 +++
 6 files changed, 110 insertions(+), 41 deletions(-)

diff --git a/ovn/controller/ofctrl.c b/ovn/controller/ofctrl.c
index c20f44c..fbd8749 100644
--- a/ovn/controller/ofctrl.c
+++ b/ovn/controller/ofctrl.c
@@ -362,6 +362,8 @@ run_S_CLEAR_FLOWS(void)
 queue_msg(encode_group_mod());
 ofputil_uninit_group_mod();
 
+force_full_process();
+
 /* Clear existing groups, to match the state of the switch. */
 if (groups) {
 ovn_group_table_clear(groups, true);
diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c
index 414e6b5..7acea2c 100644
--- a/ovn/controller/ovn-controller.c
+++ b/ovn/controller/ovn-controller.c
@@ -314,6 +314,27 @@ static struct hmap patched_datapaths = 
HMAP_INITIALIZER(_datapaths);
  * in binding.c, but consumed elsewhere. */
 static struct sset all_lports = SSET_INITIALIZER(_lports);
 
+/* Contains mapping of localvifs to OF ports for detecting if the physical
+ * configuration of the switch has changed.  This is kept independently of
+ * the same variable name in the physical module as the two are used for
+ * different purposes. */
+static struct simap localvif_to_ofport =
+SIMAP_INITIALIZER(_to_ofport);
+
+/* Contains the list of known tunnels.  This is kept independently of
+ * the same variable name in the physical module as the two are used for
+ * different purposes. */
+static struct hmap tunnels = HMAP_INITIALIZER();
+
+/* The last sequence number seen from the southbound IDL. */
+static unsigned int seqno = 0;
+
+void
+force_full_process(void)
+{
+seqno = 0;
+}
+
 int
 main(int argc, char *argv[])
 {
@@ -430,39 +451,51 @@ main(int argc, char *argv[])
 _lports);
 }
 
+enum mf_field_id mff_ovn_geneve;
+bool physical_change = true;
 if (br_int && chassis_id) {
+mff_ovn_geneve = ofctrl_run(br_int);
+physical_change = detect_and_save_physical_changes(
+_to_ofport, , mff_ovn_geneve, br_int,
+chassis_id);
+unsigned int cur_seqno = ovsdb_idl_get_seqno(ovnsb_idl_loop.idl);
+ 
 patch_run(, br_int, chassis_id, _datapaths,
   _datapaths);
 
 static struct lport_index lports;
-static struct mcgroup_index mcgroups;
 lport_index_init(, ctx.ovnsb_idl);
-mcgroup_index_init(, ctx.ovnsb_idl);
-
-enum mf_field_id mff_ovn_geneve = ofctrl_run(br_int);
-
-pinctrl_run(, , br_int, chassis_id, _datapaths);
+pinctrl_run(, , br_int, chassis_id,
+_datapaths);
 update_ct_zones(_lports, _datapaths, _zones,
 ct_zone_bitmap);
 
-struct hmap flow_table = HMAP_INITIALIZER(_table);
-lflow_run(, , , _datapaths,
-  _datapaths, _table, _zones,
-  _table);
-
-physical_run(, mff_ovn_geneve,
- br_int, chassis_id, _zones, _table,
- _datapaths, _datapaths);
-
-ofctrl_put(_table, get_nb_cfg(ctx.ovnsb_idl));
-hmap_destroy(_table);
-if (ctx.ovnsb_idl_txn) {
-int64_t cur_cfg = ofctrl_get_cur_cfg();
-if (cur_cfg && cur_cfg != chassis->nb_cfg) {
-sbrec_chassis_set_nb_cfg(chassis, cur_cfg);
+
+if (physical_change || seqno < cur_seqno) {
+seqno = cur_seqno;
+
+static struct mcgroup_index mcgroups;
+mcgroup_index_init(, ctx.ovnsb_idl);
+
+struct hmap flow_table = HMAP_INITIALIZER(_table);
+lflow_run(, , , _datapaths,
+  _datapaths, _table, _zones,
+  _table);
+
+physical_run(, mff_ovn_geneve,
+ br_int, chassis_id, _zone

Re: [ovs-dev] [RFC] ovn: minimize the impact of a compromisedchassis

2016-08-21 Thread Ryan Moats 
"dev"  wrote on 08/21/2016 03:02:12 PM:

> From: Justin Pettit 
> To: Russell Bryant 
> Cc: ovs dev 
> Date: 08/21/2016 07:30 PM
> Subject: Re: [ovs-dev] [RFC] ovn: minimize the impact of a compromised
chassis
> Sent by: "dev" 
>
>
> > On Aug 16, 2016, at 6:52 AM, Russell Bryant  wrote:
> >
> > ​Thanks for starting this discussion.​
> >
> > ​I do think making ovn-controller a read-only client of the database
seems
> > like the simplest path forward.  We should pursue that until we hit a
> > strong reason not to.
> >
> > One of the biggest questions remaining for me is how we deal with
backwards
> > compatibility.  Whatever we do here will have to account for what
happens
> > for environments running OVN from OVS 2.6 when they upgrade.
> >
> > Perhaps the most straight forward way to do that is to make this new
more
> > secure mode optional and off by default.  The downside is that it makes
the
> > system more complex, since it will have different modes it can work.
>
> I think we should avoid providing two modes if at all possible.  As
> you mentioned, it increases the complexity, which will likely make
> it more difficult to test thoroughly and deploy consistently.
>
> > An alternative would be to provide some tooling to assist with the
upgrade:
> >
> > - Document the new requirements for creating Chassis and Encap rows
> > manually
> >
> > - Provide an upgrade tool (via ovn-nbctl/ovn-sbctl/something-else) that
> > will add the "chassis" option to logical ports set based on where ports
are
> > currently bound in the southbound database.
> >
> > ​ - Provide an upgrade tool that converts the MAC_Binding table to
whatever
> > the new chassis local storage for that information would be.
>
> I think the tool approach is better.  I imagine people deploying 2.6
> will be pretty closely tied to the community still, so walking them
> through an upgrade shouldn't be too bad--especially if we make it
> easy.  Obviously, we'll try to avoid the need for that again in the
future.
>
> --Justin

I agree that the ovn-controller process should be limited to read-only
for the following tables
 SB_Global
 Logical_Flow
 Multicast_Group
 Datapath_Binding
 Address_Set
 DHCP_Options
 DHCPv6_Options

However, I'm going to argue that the suggestions for making ovn-controller
read-only for

 Chassis
 Encap
 Port_Binding
 MAC_Binding

need some more discussion as I don't think all of the suggestions to date
are entirely feasible...

First, putting the responsibility for updating the Chassis and Encap tables
on the CMS require (a) that the CMS have the ability to provide that
information
and (b) that the CMS have an OVN plugin.  I see this as moving OVN out of
the
pure networking space (at least with respect to OpenStack) as I don't see
Neutron getting this functionality any time soon.  I agree with Justin that
the right approach is to provide documentation for how to add/remove/update
entries via ovn-sbctl and then leave it to operators to incorporate that
tooling into their deployment scenarios.

For Port_Binding, while it is possible to get that information from the
CMS,
I worry (at least in the OpenStack case) about a potential race condition
during instance boot - we've already seen cases during scale testing where
the
current method (having the chassis update the port binding) doesn't
percolate
through Neutron back to Nova fast enough, and using the CMS to set the
port binding will add another half round trip of delay.  Couple that with
the
needed modifications of ovn-controller, and I think that needs some more
thought before we decide on how to change this.

MAC_Binding is a bit tricky - the problem here is how to deal where dynamic
MAC bindings need to be transferred from one chassis to another for either
HA or live migration scenarios. My preference here is to leave this alone
(i.e. allow ovn-controller to continue to write this table) and see what we
can apply from various anti-arp cache poisoning technologies to either the
IDL
or ovsdb-server itself.
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] incremental updates (was: Re: [PATCH 2/2] ovn/TODO: Add items proposed for 2.7 in OVN IRC meeting.)

2016-08-20 Thread Ryan Moats 


Ben Pfaff <b...@ovn.org> wrote on 08/19/2016 02:36:42 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: ovs dev <dev@openvswitch.org>, Numan Siddique <nusid...@redhat.com>
> Date: 08/19/2016 02:37 PM
> Subject: Re: incremental updates (was: Re: [ovs-dev] [PATCH 2/2]
> ovn/TODO: Add items proposed for 2.7 in OVN IRC meeting.)
>
> On Fri, Aug 19, 2016 at 10:24:21AM -0500, Ryan Moats wrote:
> > Ben Pfaff <b...@ovn.org> wrote on 08/19/2016 10:11:59 AM:
> >
> > > From: Ben Pfaff <b...@ovn.org>
> > > To: Ryan Moats/Omaha/IBM@IBMUS
> > > Cc: Numan Siddique <nusid...@redhat.com>, ovs dev
<dev@openvswitch.org>
> > > Date: 08/19/2016 10:12 AM
> > > Subject: incremental updates (was: Re: [ovs-dev] [PATCH 2/2] ovn/
> > > TODO: Add items proposed for 2.7 in OVN IRC meeting.)
> > >
> > > On Fri, Aug 19, 2016 at 08:34:45AM -0500, Ryan Moats wrote:
> > > > We've been running incremental processing in ovn-controller here
for
> > quite
> > > > a while (even before it merged officially) and given our
experience,
> > > > I've been doing some hard thinking about it.
> > > >
> > > > The original goal of the patch set was to allow ovn-controller to
use
> > > > incremental updates as much as possible.  However, the range of
> > possible
> > > > changes in inputs resulted in ovn-controller having to continue to
> > > > maintain the ability to run a full update in those cases where
> > previously
> > > > calculated rules are no longer correct.
> > > >
> > > > What we've found is that most configuration events from our CMS
> > (Neutron)
> > > > end up triggering a full ovn-controller update, and so the end
result
> > of
> > > > the incremental processing code is not that ovn-controller is doing
> > > > incremental updates most of the time, but rather that
ovn-controller
> > > > doesn't recalculate changes in-between modifications.
> > > >
> > > > While we still feel that the above is a win, I'm coming to the
> > conclusion
> > > > that the current code base has added unnecessary complexity to
achieve
> > > > this. Based on this, I'm thinking of the following approach:
> > > >
> > > > 1) going back to doing full processing every cycle, while still
> > > > keeping the persistence of items where we can, because I feel that
> > > > persistence has allowed us to handle cases where we need to skip a
poll
> > > > cycle that we didn't have before and that has improved things.
> > >
> > > What I'm hearing here is that we'll do full processing if any
processing
> > > at all is necessary.  Seems fine to me.
> > >
> > > > 2) introducing a new command flag to allow those that don't want to
> > > > run in what I'm now calling quiet mode to continue to do full
> > processing
> > > > every cycle. (In retrospect, I should have proposed this up front
for
> > i-p
> > > > to allow for better isolation of that code, but as they say,
hindsight
> > > > is always 20/20).
> > >
> > > Hmm.  I would hope that "Full processing if any processing" is fairly
> > > easy to get right.  However, I'm OK with having a flag as a
fail-safe.
> > > In time, I would hope that we could delete the flag.
> > >
> > > > 3) For quiet mode, check the integration bridge and Ben's sequence
> > > > number information in the SB database to determine if anything has
> > > > changed since the last cycle. If something has changed, run the
full
> > > > processing code. If not, quiesce for a poll cycle.
> > >
> > > I don't think that the sequence numbers are a good way to determine
> > > whether something has changed.  Without --wait, for example,
ovn-nbctl
> > > doesn't change the sequence number.
> >
> > Yes on #1 - that's the intent.
> >
> > Apologies for not being more clear that in re my thinking for
suggesting
> > the command line flag.  I agree that we'd deprecate it long term if we
> > added it, but  (and now I move to #3)
> >
> > I know that sequence numbers are only there today if asked for, and I
was
> > thinking of arguing that they should always be there and that ovn-nbctl
> > should *check* them only if --wait is set.  However, there might be
> > other ways to figure out if the SB DBs have changed (I'll admit, I've
not
> > thought of any yet - anybody have any ideas?)
>
> The IDL has its own sequence number for that, see ovsdb_idl_get_seqno().
>

Yep, that helped turn the trick - ran into a few other nits, so I broke the
patches in two:

http://patchwork.ozlabs.org/patch/661159/
http://patchwork.ozlabs.org/patch/661161/

The unit test cases pass for me, but it would be good if folks take a look
at both as extra eyes are always good...

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH] ovn-controller: add quiet mode

2016-08-20 Thread Ryan Moats 
As discussed in [1], what the incremental processing code
actually accomplished was that the ovn-controller would
be "quiet" and not burn CPU when things weren't changing.
This patch set recreates this state by calculating whether
changes have occured that would require a full calculation
to be performed.

This commit depends on f5d916cb ("ovn-controller:
Back out incremental processing")

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 ovn/controller/ofctrl.c |  2 ++
 ovn/controller/ovn-controller.c | 77 +
 ovn/controller/ovn-controller.h |  1 +
 ovn/controller/patch.c  |  2 ++
 ovn/controller/physical.c   | 55 +++--
 ovn/controller/physical.h   |  5 +++
 6 files changed, 101 insertions(+), 41 deletions(-)

diff --git a/ovn/controller/ofctrl.c b/ovn/controller/ofctrl.c
index c20f44c..fbd8749 100644
--- a/ovn/controller/ofctrl.c
+++ b/ovn/controller/ofctrl.c
@@ -362,6 +362,8 @@ run_S_CLEAR_FLOWS(void)
 queue_msg(encode_group_mod());
 ofputil_uninit_group_mod();
 
+force_full_process();
+
 /* Clear existing groups, to match the state of the switch. */
 if (groups) {
 ovn_group_table_clear(groups, true);
diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c
index 414e6b5..7acea2c 100644
--- a/ovn/controller/ovn-controller.c
+++ b/ovn/controller/ovn-controller.c
@@ -314,6 +314,27 @@ static struct hmap patched_datapaths = 
HMAP_INITIALIZER(_datapaths);
  * in binding.c, but consumed elsewhere. */
 static struct sset all_lports = SSET_INITIALIZER(_lports);
 
+/* Contains mapping of localvifs to OF ports for detecting if the physical
+ * configuration of the switch has changed.  This is kept independently of
+ * the same variable name in the physical module as the two are used for
+ * different purposes. */
+static struct simap localvif_to_ofport =
+SIMAP_INITIALIZER(_to_ofport);
+
+/* Contains the list of known tunnels.  This is kept independently of
+ * the same variable name in the physical module as the two are used for
+ * different purposes. */
+static struct hmap tunnels = HMAP_INITIALIZER();
+
+/* The last sequence number seen from the southbound IDL. */
+static unsigned int seqno = 0;
+
+void
+force_full_process(void)
+{
+seqno = 0;
+}
+
 int
 main(int argc, char *argv[])
 {
@@ -430,39 +451,51 @@ main(int argc, char *argv[])
 _lports);
 }
 
+enum mf_field_id mff_ovn_geneve;
+bool physical_change = true;
 if (br_int && chassis_id) {
+mff_ovn_geneve = ofctrl_run(br_int);
+physical_change = detect_and_save_physical_changes(
+_to_ofport, , mff_ovn_geneve, br_int,
+chassis_id);
+unsigned int cur_seqno = ovsdb_idl_get_seqno(ovnsb_idl_loop.idl);
+ 
 patch_run(, br_int, chassis_id, _datapaths,
   _datapaths);
 
 static struct lport_index lports;
-static struct mcgroup_index mcgroups;
 lport_index_init(, ctx.ovnsb_idl);
-mcgroup_index_init(, ctx.ovnsb_idl);
-
-enum mf_field_id mff_ovn_geneve = ofctrl_run(br_int);
-
-pinctrl_run(, , br_int, chassis_id, _datapaths);
+pinctrl_run(, , br_int, chassis_id,
+_datapaths);
 update_ct_zones(_lports, _datapaths, _zones,
 ct_zone_bitmap);
 
-struct hmap flow_table = HMAP_INITIALIZER(_table);
-lflow_run(, , , _datapaths,
-  _datapaths, _table, _zones,
-  _table);
-
-physical_run(, mff_ovn_geneve,
- br_int, chassis_id, _zones, _table,
- _datapaths, _datapaths);
-
-ofctrl_put(_table, get_nb_cfg(ctx.ovnsb_idl));
-hmap_destroy(_table);
-if (ctx.ovnsb_idl_txn) {
-int64_t cur_cfg = ofctrl_get_cur_cfg();
-if (cur_cfg && cur_cfg != chassis->nb_cfg) {
-sbrec_chassis_set_nb_cfg(chassis, cur_cfg);
+
+if (physical_change || seqno < cur_seqno) {
+seqno = cur_seqno;
+
+static struct mcgroup_index mcgroups;
+mcgroup_index_init(, ctx.ovnsb_idl);
+
+struct hmap flow_table = HMAP_INITIALIZER(_table);
+lflow_run(, , , _datapaths,
+  _datapaths, _table, _zones,
+  _table);
+
+physical_run(, mff_ovn_geneve,
+ br_int, chassis_id, _zones, _table,
+ _datapaths, _datapaths);
+
+ofctrl_put(_table, get_nb_cfg(ctx.ovnsb_idl));
+hmap_destroy(_table);
+if (ctx.ovnsb

Re: [ovs-dev] [PATCH] ovn-northd: Fix spelling error in comment.

2016-08-20 Thread Ryan Moats 


"dev" <dev-boun...@openvswitch.org> wrote on 08/20/2016 10:21:08 AM:

> From: Ben Pfaff <b...@ovn.org>
> To: dev@openvswitch.org
> Cc: Ben Pfaff <b...@ovn.org>
> Date: 08/20/2016 10:21 AM
> Subject: [ovs-dev] [PATCH] ovn-northd: Fix spelling error in comment.
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> Signed-off-by: Ben Pfaff <b...@ovn.org>
> ---
>  ovn/northd/ovn-northd.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/ovn/northd/ovn-northd.c b/ovn/northd/ovn-northd.c
> index 625937d..0874a9c 100644
> --- a/ovn/northd/ovn-northd.c
> +++ b/ovn/northd/ovn-northd.c
> @@ -1921,7 +1921,7 @@ build_pre_acls(struct ovn_datapath *od, struct
> hmap *lflows,
>  ovn_lflow_add(lflows, od, S_SWITCH_IN_PRE_ACL, 0, "1", "next;");
>  ovn_lflow_add(lflows, od, S_SWITCH_OUT_PRE_ACL, 0, "1", "next;");
>
> -/* If there are any stateful ACL rules in this dapapath, we must
> +/* If there are any stateful ACL rules in this datapath, we must
>   * send all IP packets through the conntrack action, which handles
>   * defragmentation, in order to match L4 headers. */
>  if (has_stateful) {

And would you believe it took me 3 readings to *find* it??? (#sadpanda)

Acked-by: Ryan Moats <rmo...@us.ibm.com>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH] ovn-controller: Back out incremental processing

2016-08-20 Thread Ryan Moats 
As [1] indicates, incremental processing hasn't resulted
in an improvement worth the complexity it has added.
This patch backs out all of the code specific to incremental
processing, along with the persisting of OF flows,
logical ports and multicast groups.

[1] http://openvswitch.org/pipermail/dev/2016-August/078272.html

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 Note: while this passes all of the current ovn unit and e2e
 tests, I'm not 100% sure it is fully correct as a fair number
 of the reported issues are not captured in the test suite.

 ovn/controller/binding.c| 120 +---
 ovn/controller/binding.h|   1 -
 ovn/controller/encaps.c | 111 ++-
 ovn/controller/lflow.c  | 101 -
 ovn/controller/lflow.h  |   4 +-
 ovn/controller/lport.c  | 220 +
 ovn/controller/lport.h  |  24 +---
 ovn/controller/ofctrl.c | 306 +---
 ovn/controller/ofctrl.h |  16 +--
 ovn/controller/ovn-controller.c |  26 ++--
 ovn/controller/patch.c  |   6 -
 ovn/controller/physical.c   | 166 ++
 ovn/controller/physical.h   |   3 +-
 13 files changed, 281 insertions(+), 823 deletions(-)

diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
index c26007d..f2552fa 100644
--- a/ovn/controller/binding.c
+++ b/ovn/controller/binding.c
@@ -33,15 +33,6 @@ VLOG_DEFINE_THIS_MODULE(binding);
 /* A set of the iface-id values of local interfaces on this chassis. */
 static struct sset local_ids = SSET_INITIALIZER(_ids);
 
-/* When this gets set to true, the next run will re-check all binding records. 
*/
-static bool process_full_binding = false;
-
-void
-binding_reset_processing(void)
-{
-process_full_binding = true;
-}
-
 void
 binding_register_ovs_idl(struct ovsdb_idl *ovs_idl)
 {
@@ -139,7 +130,6 @@ remove_local_datapath(struct hmap *local_datapaths, struct 
local_datapath *ld)
 }
 hmap_remove(local_datapaths, >hmap_node);
 free(ld);
-lflow_reset_processing();
 }
 
 static void
@@ -156,9 +146,6 @@ add_local_datapath(struct hmap *local_datapaths,
 memcpy(>uuid, _rec->header_.uuid, sizeof ld->uuid);
 hmap_insert(local_datapaths, >hmap_node,
 binding_rec->datapath->tunnel_key);
-lport_index_reset();
-mcgroup_index_reset();
-lflow_reset_processing();
 }
 
 static void
@@ -268,80 +255,49 @@ binding_run(struct controller_ctx *ctx, const struct 
ovsrec_bridge *br_int,
 }
 
 if (br_int) {
-if (ctx->ovnsb_idl_txn && get_local_iface_ids(br_int, _to_iface,
-  all_lports)) {
-process_full_binding = true;
-}
-} else {
-/* We have no integration bridge, therefore no local logical ports.
- * We'll remove our chassis from all port binding records below. */
-process_full_binding = true;
+get_local_iface_ids(br_int, _to_iface, all_lports);
 }
 
-/* Run through each binding record to see if it is resident on this
- * chassis and update the binding accordingly.  This includes both
- * directly connected logical ports and children of those ports. */
-if (process_full_binding) {
-/* Detect any entries in all_lports that have been deleted.
- * In particular, this will catch localnet ports that we
- * put in all_lports. */
-struct sset removed_lports = SSET_INITIALIZER(_lports);
-sset_clone(_lports, all_lports);
-
-struct hmap keep_local_datapath_by_uuid =
-HMAP_INITIALIZER(_local_datapath_by_uuid);
-SBREC_PORT_BINDING_FOR_EACH(binding_rec, ctx->ovnsb_idl) {
-sset_find_and_delete(_lports, binding_rec->logical_port);
-consider_local_datapath(ctx, chassis_rec, binding_rec,
-local_datapaths, _to_iface,
-all_lports);
-struct local_datapath *ld = xzalloc(sizeof *ld);
-memcpy(>uuid, _rec->header_.uuid, sizeof ld->uuid);
-hmap_insert(_local_datapath_by_uuid, >uuid_hmap_node,
-uuid_hash(>uuid));
-}
-struct local_datapath *old_ld, *next;
-HMAP_FOR_EACH_SAFE (old_ld, next, hmap_node, local_datapaths) {
-if (!local_datapath_lookup_by_uuid(_local_datapath_by_uuid,
-   _ld->uuid)) {
-remove_local_datapath(local_datapaths, old_ld);
-}
-}
-struct local_datapath *ld;
-HMAP_FOR_EACH_SAFE (ld, next, uuid_hmap_node,
-_local_datapath_by_uuid) {
-hmap_remove(_local_datapath_by_uuid, >uuid_hmap_node);
-free(ld);
-}
-hmap_destroy(_local_datapath_by_uuid);
+/* Detect any en

Re: [ovs-dev] another possible cause of incremental update problems

2016-08-20 Thread Ryan Moats 


Ben Pfaff <b...@ovn.org> wrote on 08/20/2016 10:32:50 AM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: ovs dev <dev@openvswitch.org>
> Date: 08/20/2016 10:33 AM
> Subject: another possible cause of incremental update problems
>
> There are additional possible causes of incremental update problems that
> I haven't previously seen mentioned here.  If you look at the functions
> that implement incremental update, they tend to start with:
>
> if (!ctx->ovnsb_idl_txn) {
> return NULL;
> }
>
> or similar.  This makes sense, because there is no point in composing an
> update if it can't be committed.  However, because the main loop call
> ovsdb_idl_txn_clear() on every iteration, that means that any updates
> from this iteration will be lost.
>
> Also, transactions can fail.  In that case, the transaction needs to be
> recomposed and retried.  ovn-controller currently doesn't have any
> mechanism for that.

Yep, there are all sorts of interesting failure modes, but the key point
I was looking to make with the analysis was that *even if everything
worked perfectly* the code doesn't buy us what I hoped it would buy when
I started - it ended up in a different place where the tradeoff no
longer makes sense to me... (c'est la vie)

As a side note, I have the revert change done and I'll be sending that
out shortly. The quiet mode change isn't passing all of the current unit
test cases (yet), so I'm still looking at it.  I plan on sending it out
on Monday, either as a real change if I figure out what's going wrong
in the unit tests or as an RFC if I don't...


___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] ovn-controller: Fix memory leak of keep_local_datapath_by_uuid.

2016-08-19 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 08/19/2016 10:50:17 AM:

> From: Ben Pfaff <b...@ovn.org>
> To: dev@openvswitch.org
> Cc: Ben Pfaff <b...@ovn.org>, Ryan Moats/Omaha/IBM@IBMUS, Ramu
> Ramamurthy <ramu.ramamur...@gmail.com>
> Date: 08/19/2016 10:50 AM
> Subject: [PATCH] ovn-controller: Fix memory leak of
> keep_local_datapath_by_uuid.
>
> Fixes: 263064aeaa31 ("Convert binding_run to incremental processing.")
> Reported-by: Ramu Ramamurthy <ramu.ramamur...@gmail.com>
> CC: RYAN D. MOATS <rmo...@us.ibm.com>
> Signed-off-by: Ben Pfaff <b...@ovn.org>
> ---
>  ovn/controller/binding.c | 6 ++
>  1 file changed, 6 insertions(+)
>
> diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c
> index 5f03702..c26007d 100644
> --- a/ovn/controller/binding.c
> +++ b/ovn/controller/binding.c
> @@ -307,6 +307,12 @@ binding_run(struct controller_ctx *ctx, const
> struct ovsrec_bridge *br_int,
>  remove_local_datapath(local_datapaths, old_ld);
>  }
>  }
> +struct local_datapath *ld;
> +HMAP_FOR_EACH_SAFE (ld, next, uuid_hmap_node,
> +_local_datapath_by_uuid) {
> +hmap_remove(_local_datapath_by_uuid, >
uuid_hmap_node);
> +free(ld);
> +}
>  hmap_destroy(_local_datapath_by_uuid);
>
>      /* Any remaining entries in removed_lports are logical ports
that
> --
> 2.1.3
>

Even though this is likely going away very soon --

Acked-by: Ryan Moats <rmo...@us.ibm.com>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] incremental updates (was: Re: [PATCH 2/2] ovn/TODO: Add items proposed for 2.7 in OVN IRC meeting.)

2016-08-19 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 08/19/2016 10:11:59 AM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: Numan Siddique <nusid...@redhat.com>, ovs dev <dev@openvswitch.org>
> Date: 08/19/2016 10:12 AM
> Subject: incremental updates (was: Re: [ovs-dev] [PATCH 2/2] ovn/
> TODO: Add items proposed for 2.7 in OVN IRC meeting.)
>
> On Fri, Aug 19, 2016 at 08:34:45AM -0500, Ryan Moats wrote:
> > We've been running incremental processing in ovn-controller here for
quite
> > a while (even before it merged officially) and given our experience,
> > I've been doing some hard thinking about it.
> >
> > The original goal of the patch set was to allow ovn-controller to use
> > incremental updates as much as possible.  However, the range of
possible
> > changes in inputs resulted in ovn-controller having to continue to
> > maintain the ability to run a full update in those cases where
previously
> > calculated rules are no longer correct.
> >
> > What we've found is that most configuration events from our CMS
(Neutron)
> > end up triggering a full ovn-controller update, and so the end result
of
> > the incremental processing code is not that ovn-controller is doing
> > incremental updates most of the time, but rather that ovn-controller
> > doesn't recalculate changes in-between modifications.
> >
> > While we still feel that the above is a win, I'm coming to the
conclusion
> > that the current code base has added unnecessary complexity to achieve
> > this. Based on this, I'm thinking of the following approach:
> >
> > 1) going back to doing full processing every cycle, while still
> > keeping the persistence of items where we can, because I feel that
> > persistence has allowed us to handle cases where we need to skip a poll
> > cycle that we didn't have before and that has improved things.
>
> What I'm hearing here is that we'll do full processing if any processing
> at all is necessary.  Seems fine to me.
>
> > 2) introducing a new command flag to allow those that don't want to
> > run in what I'm now calling quiet mode to continue to do full
processing
> > every cycle. (In retrospect, I should have proposed this up front for
i-p
> > to allow for better isolation of that code, but as they say, hindsight
> > is always 20/20).
>
> Hmm.  I would hope that "Full processing if any processing" is fairly
> easy to get right.  However, I'm OK with having a flag as a fail-safe.
> In time, I would hope that we could delete the flag.
>
> > 3) For quiet mode, check the integration bridge and Ben's sequence
> > number information in the SB database to determine if anything has
> > changed since the last cycle. If something has changed, run the full
> > processing code. If not, quiesce for a poll cycle.
>
> I don't think that the sequence numbers are a good way to determine
> whether something has changed.  Without --wait, for example, ovn-nbctl
> doesn't change the sequence number.

Yes on #1 - that's the intent.

Apologies for not being more clear that in re my thinking for suggesting
the command line flag.  I agree that we'd deprecate it long term if we
added it, but  (and now I move to #3)

I know that sequence numbers are only there today if asked for, and I was
thinking of arguing that they should always be there and that ovn-nbctl
should *check* them only if --wait is set.  However, there might be
other ways to figure out if the SB DBs have changed (I'll admit, I've not
thought of any yet - anybody have any ideas?)
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH 2/2] ovn/TODO: Add items proposed for 2.7 in OVN IRC meeting.

2016-08-19 Thread Ryan Moats 


"dev"  wrote on 08/19/2016 12:07:36 AM:

> From: Numan Siddique 
> To: Ben Pfaff 
> Cc: ovs dev 
> Date: 08/19/2016 12:07 AM
> Subject: Re: [ovs-dev] [PATCH 2/2] ovn/TODO: Add items proposed for
> 2.7 in OVN IRC meeting.
> Sent by: "dev" 
>
> On Thu, Aug 18, 2016 at 11:44 PM, Ben Pfaff  wrote:
>
> > Signed-off-by: Ben Pfaff 
> > ---
> >  ovn/TODO | 55 +++
> >  1 file changed, 55 insertions(+)
> >
> > diff --git a/ovn/TODO b/ovn/TODO
> > index b3c4831..97a5fc9 100644
> > --- a/ovn/TODO
> > +++ b/ovn/TODO
> > @@ -1,5 +1,60 @@
> >  -*- outline -*-
> >
> > +* Work out database for clustering or HA properly.
> > +
> > +* Compromised chassis mitigation.
> > +
> > +Possibly depends on database solution.
> > +
> > +* Get incremental updates in ovn-controller and ovn-northd in some
> > +  sensible way.
> > +
> >
>
> ​May be the below can also be added which you mentioned in the IRC meeting
> (11th August).
>
> --
> blp regXboi: We should eventually make OVN use OpenFlow "bundles" so that
> it transactionally replaces the flow table instead of deleting and then
> repopulating it.
> ​
>
> ​-​

We've been running incremental processing in ovn-controller here for quite
a while (even before it merged officially) and given our experience,
I've been doing some hard thinking about it.

The original goal of the patch set was to allow ovn-controller to use
incremental updates as much as possible.  However, the range of possible
changes in inputs resulted in ovn-controller having to continue to
maintain the ability to run a full update in those cases where previously
calculated rules are no longer correct.

What we've found is that most configuration events from our CMS (Neutron)
end up triggering a full ovn-controller update, and so the end result of
the incremental processing code is not that ovn-controller is doing
incremental updates most of the time, but rather that ovn-controller
doesn't recalculate changes in-between modifications.

While we still feel that the above is a win, I'm coming to the conclusion
that the current code base has added unnecessary complexity to achieve
this. Based on this, I'm thinking of the following approach:

1) going back to doing full processing every cycle, while still
keeping the persistence of items where we can, because I feel that
persistence has allowed us to handle cases where we need to skip a poll
cycle that we didn't have before and that has improved things.
2) introducing a new command flag to allow those that don't want to
run in what I'm now calling quiet mode to continue to do full processing
every cycle. (In retrospect, I should have proposed this up front for i-p
to allow for better isolation of that code, but as they say, hindsight
is always 20/20).
3) For quiet mode, check the integration bridge and Ben's sequence
number information in the SB database to determine if anything has
changed since the last cycle. If something has changed, run the full
processing code. If not, quiesce for a poll cycle.

Thoughts?
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [ovs-dev, v5] Add read-only option to ovs-dpctl and ovs-ofctl commands.

2016-08-16 Thread Ryan Moats 

Ben Pfaff <b...@ovn.org> wrote on 08/15/2016 07:28:18 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/16/2016 01:54 AM
> Subject: Re: [ovs-dev, v5] Add read-only option to ovs-dpctl and
> ovs-ofctl commands.
>
> On Mon, Aug 15, 2016 at 06:47:29PM +, Ryan Moats wrote:
> > ovs-dpctl and ovs-ofctl lack a read-only option to prevent
> > running of commands that perform read-write operations.  Add
> > it and the necessary scaffolding to each.
> >
> > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
>
> I folded in a few changes, see below, and applied this to master.
>
> --8<--cut here-->8--
>
> diff --git a/lib/command-line.c b/lib/command-line.c
> index 92ce888..8128331 100644
> --- a/lib/command-line.c
> +++ b/lib/command-line.c
> @@ -87,11 +87,10 @@ ovs_cmdl_print_options(const struct option options[])
>  ds_destroy();
>  }
>
> -/* Whether to commit changes or not. */
> -static bool read_only;
> -
>  static void
> -_ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const struct
> ovs_cmdl_command commands[])
> +ovs_cmdl_run_command__(struct ovs_cmdl_context *ctx,
> +   const struct ovs_cmdl_command commands[],
> +   bool read_only)
>  {
>  const struct ovs_cmdl_command *p;
>
> @@ -144,16 +143,14 @@ void
>  ovs_cmdl_run_command(struct ovs_cmdl_context *ctx,
>   const struct ovs_cmdl_command commands[])
>  {
> -read_only = false;
> -_ovs_cmdl_run_command(ctx, commands);
> +ovs_cmdl_run_command__(ctx, commands, false);
>  }
>
>  void
>  ovs_cmdl_run_command_read_only(struct ovs_cmdl_context *ctx,
> const struct ovs_cmdl_command commands[])
>  {
> -read_only = true;
> -_ovs_cmdl_run_command(ctx, commands);
> +ovs_cmdl_run_command__(ctx, commands, true);
>  }
>
>  /* Process title. */
> diff --git a/tests/test-ovsdb.c b/tests/test-ovsdb.c
> index c123e43..35b70e9 100644
> --- a/tests/test-ovsdb.c
> +++ b/tests/test-ovsdb.c
> @@ -2720,7 +2720,7 @@ static struct ovs_cmdl_command all_commands[] = {
>  { "log-io", NULL, 2, INT_MAX, do_log_io, OVS_RO },
>  { "default-atoms", NULL, 0, 0, do_default_atoms, OVS_RO },
>  { "default-data", NULL, 0, 0, do_default_data, OVS_RO },
> -{ "diff-data", NULL, 3, INT_MAX, do_diff_data},
> +{ "diff-data", NULL, 3, INT_MAX, do_diff_data, OVS_RO },
>  { "parse-atomic-type", NULL, 1, 1, do_parse_atomic_type, OVS_RO },
>  { "parse-base-type", NULL, 1, 1, do_parse_base_type, OVS_RO },
>  { "parse-type", NULL, 1, 1, do_parse_type, OVS_RO },
>

The first two changes work for me -

The last one annoys me because I *know* I fixed that as compiling fails
without it.  So, now I'm wondering how the heck it reverted in the patch
set...

It's all good and I'll send out the more controversial patch later
today...
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v5] Add read-only option to ovs-dpctl and ovs-ofctl commands.

2016-08-15 Thread Ryan Moats 
ovs-dpctl and ovs-ofctl lack a read-only option to prevent
running of commands that perform read-write operations.  Add
it and the necessary scaffolding to each.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 lib/command-line.c |  51 +
 lib/command-line.h |   6 +-
 lib/db-ctl-base.h  |   2 +-
 lib/dpctl.c|  47 ++--
 lib/dpctl.h|   3 +
 ovsdb/ovsdb-tool.c |  28 
 tests/ovstest.c|   4 +-
 tests/test-bitmap.c|   6 +-
 tests/test-ccmap.c |   6 +-
 tests/test-classifier.c|  28 
 tests/test-cmap.c  |   6 +-
 tests/test-conntrack.c |   6 +-
 tests/test-heap.c  |  14 ++--
 tests/test-jsonrpc.c   |  10 +--
 tests/test-netlink-conntrack.c |   8 +--
 tests/test-ovn.c   |  26 +++
 tests/test-ovsdb.c |  80 ++---
 tests/test-reconnect.c |  32 -
 tests/test-util.c  |  34 -
 tests/test-vconn.c |  18 ++---
 utilities/ovs-dpctl.c  |   7 ++
 utilities/ovs-ofctl.8.in   |   3 +
 utilities/ovs-ofctl.c  | 160 ++---
 23 files changed, 323 insertions(+), 262 deletions(-)

diff --git a/lib/command-line.c b/lib/command-line.c
index bda5ed6..92ce888 100644
--- a/lib/command-line.c
+++ b/lib/command-line.c
@@ -87,20 +87,11 @@ ovs_cmdl_print_options(const struct option options[])
 ds_destroy();
 }
 
-/* Runs the command designated by argv[0] within the command table specified by
- * 'commands', which must be terminated by a command whose 'name' member is a
- * null pointer.
- *
- * Command-line options should be stripped off, so that a typical invocation
- * looks like:
- *struct ovs_cmdl_context ctx = {
- *.argc = argc - optind,
- *.argv = argv + optind,
- *};
- *ovs_cmdl_run_command(, my_commands);
- * */
-void
-ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const struct 
ovs_cmdl_command commands[])
+/* Whether to commit changes or not. */
+static bool read_only;
+
+static void
+_ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const struct 
ovs_cmdl_command commands[])
 {
 const struct ovs_cmdl_command *p;
 
@@ -118,6 +109,10 @@ ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const 
struct ovs_cmdl_command
 VLOG_FATAL("'%s' command takes at most %d arguments",
p->name, p->max_args);
 } else {
+if (p->mode == OVS_RW && read_only) {
+VLOG_FATAL("'%s' command does not work in read only mode",
+   p->name);
+}
 p->handler(ctx);
 if (ferror(stdout)) {
 VLOG_FATAL("write to stdout failed");
@@ -132,6 +127,34 @@ ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const 
struct ovs_cmdl_command
 
 VLOG_FATAL("unknown command '%s'; use --help for help", ctx->argv[0]);
 }
+
+/* Runs the command designated by argv[0] within the command table specified by
+ * 'commands', which must be terminated by a command whose 'name' member is a
+ * null pointer.
+ *
+ * Command-line options should be stripped off, so that a typical invocation
+ * looks like:
+ *struct ovs_cmdl_context ctx = {
+ *.argc = argc - optind,
+ *.argv = argv + optind,
+ *};
+ *ovs_cmdl_run_command(, my_commands);
+ * */
+void
+ovs_cmdl_run_command(struct ovs_cmdl_context *ctx,
+ const struct ovs_cmdl_command commands[])
+{
+read_only = false;
+_ovs_cmdl_run_command(ctx, commands);
+}
+
+void
+ovs_cmdl_run_command_read_only(struct ovs_cmdl_context *ctx,
+   const struct ovs_cmdl_command commands[])
+{
+read_only = true;
+_ovs_cmdl_run_command(ctx, commands);
+}
 
 /* Process title. */
 
diff --git a/lib/command-line.h b/lib/command-line.h
index e9e3b7b..00ace94 100644
--- a/lib/command-line.h
+++ b/lib/command-line.h
@@ -41,12 +41,16 @@ struct ovs_cmdl_command {
 int min_args;
 int max_args;
 ovs_cmdl_handler handler;
+enum { OVS_RO, OVS_RW } mode;/* Does this command modify things? */
 };
 
 char *ovs_cmdl_long_options_to_short_options(const struct option *options);
 void ovs_cmdl_print_options(const struct option *options);
 void ovs_cmdl_print_commands(const struct ovs_cmdl_command *commands);
-void ovs_cmdl_run_command(struct ovs_cmdl_context *, const struct 
ovs_cmdl_command[]);
+void ovs_cmdl_run_command(struct ovs_cmdl_context *,
+  const struct ovs_cmdl_command[]);
+void ovs_cmdl_run_command_read_only(struct ovs_cmdl_context *,
+const struct ovs_cmdl_command[]);
 
 void ovs_cmdl_proctitle_init(int argc, char **argv);
 #if defined(__FreeB

Re: [ovs-dev] [ovs-dev, v2] ovn-controller: Remove flows created for now deleted SB database rows.

2016-08-14 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 08/14/2016 10:06:41 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/14/2016 10:06 PM
> Subject: Re: [ovs-dev, v2] ovn-controller: Remove flows created for
> now deleted SB database rows.
>
> On Fri, Aug 12, 2016 at 09:35:07AM -0500, Ryan Moats wrote:
> > Ensure that rows created for deleted port binding and
> > multicast group rows are cleared when doing full processing.
> >
> > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> > ---
> > v1->v2:
> >   - replace use of ssets for storing UUIDs as strings with hmaps for
> > storing the UUID structs themselves
> >   - include an optimization suggested by Ben when reviewing v1
>
> I'm pretty sure that this leaks all of the nodes removed from the hmap
> by hmap_clear(new) in rationalize_hmap_and_delete_flows().
>

Oh, am I red-faced!

The comment after the new commit message in the new PS says it all...
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v3] ovn-controller: Remove flows created for now deleted SB database rows.

2016-08-14 Thread Ryan Moats 
Ensure that rows created for deleted port binding and
multicast group rows are cleared when doing full processing.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
v2->v3:
  - correctly clear old hmap (that's what I get for not reading the hmap_clear
code first)

v1->v2:
  - replace use of ssets for storing UUIDs as strings with hmaps for
storing the UUID structs themselves
  - include an optimization suggested by Ben when reviewing v1

 ovn/controller/physical.c | 59 ++-
 1 file changed, 58 insertions(+), 1 deletion(-)

diff --git a/ovn/controller/physical.c b/ovn/controller/physical.c
index 4448756..0bd228c 100644
--- a/ovn/controller/physical.c
+++ b/ovn/controller/physical.c
@@ -20,6 +20,7 @@
 #include "lflow.h"
 #include "lib/poll-loop.h"
 #include "ofctrl.h"
+#include "openvswitch/hmap.h"
 #include "openvswitch/match.h"
 #include "openvswitch/ofp-actions.h"
 #include "openvswitch/ofpbuf.h"
@@ -58,6 +59,15 @@ static struct simap localvif_to_ofport =
 SIMAP_INITIALIZER(_to_ofport);
 static struct hmap tunnels = HMAP_INITIALIZER();
 
+struct uuid_hash_node {
+struct hmap_node node;
+struct uuid uuid;
+};
+
+static struct hmap port_binding_uuids = HMAP_INITIALIZER(_binding_uuids);
+static struct hmap multicast_group_uuids =
+HMAP_INITIALIZER(_group_uuids);
+
 /* UUID to identify OF flows not associated with ovsdb rows. */
 static struct uuid *hc_uuid = NULL;
 static bool full_binding_processing = false;
@@ -636,6 +646,35 @@ consider_mc_group(enum mf_field_id mff_ovn_geneve,
 sset_destroy(_chassis);
 }
 
+static bool
+find_uuid_in_hmap(struct hmap *hmap_p, struct uuid *uuid)
+{
+struct uuid_hash_node *candidate;
+HMAP_FOR_EACH_WITH_HASH (candidate, node, uuid_hash(uuid), hmap_p) {
+if (uuid_equals(uuid, >uuid)) {
+return true;
+}
+}
+return false;
+}
+
+/* Deletes the flows whose UUIDs are in 'old' but not 'new', and then replaces
+ * 'old' by 'new'. */
+static void
+rationalize_hmap_and_delete_flows(struct hmap *old, struct hmap *new)
+{
+struct uuid_hash_node *uuid_node, *old_node;
+HMAP_FOR_EACH_SAFE (uuid_node, old_node, node, old) {
+if (!find_uuid_in_hmap(new, _node->uuid)) {
+ofctrl_remove_flows(_node->uuid);
+}
+}
+hmap_swap(old, new);
+HMAP_FOR_EACH_POP(uuid_node, node, new) {
+free(uuid_node);
+}
+}
+
 void
 physical_run(struct controller_ctx *ctx, enum mf_field_id mff_ovn_geneve,
  const struct ovsrec_bridge *br_int, const char *this_chassis_id,
@@ -791,6 +830,8 @@ physical_run(struct controller_ctx *ctx, enum mf_field_id 
mff_ovn_geneve,
  * 64 for logical-to-physical translation. */
 const struct sbrec_port_binding *binding;
 if (full_binding_processing) {
+struct hmap new_port_binding_uuids =
+HMAP_INITIALIZER(_port_binding_uuids);
 SBREC_PORT_BINDING_FOR_EACH (binding, ctx->ovnsb_idl) {
 /* Because it is possible in the above code to enter this
  * for loop without having cleared the flow table first, we
@@ -798,7 +839,14 @@ physical_run(struct controller_ctx *ctx, enum mf_field_id 
mff_ovn_geneve,
 ofctrl_remove_flows(>header_.uuid);
 consider_port_binding(mff_ovn_geneve, ct_zones, local_datapaths,
   patched_datapaths, binding, );
-}
+struct uuid_hash_node *hash_node = xzalloc(sizeof *hash_node);
+hash_node->uuid = binding->header_.uuid;
+hmap_insert(_port_binding_uuids, _node->node,
+uuid_hash(_node->uuid));
+}
+rationalize_hmap_and_delete_flows(_binding_uuids,
+  _port_binding_uuids);
+hmap_destroy(_port_binding_uuids);
 full_binding_processing = false;
 } else {
 SBREC_PORT_BINDING_FOR_EACH_TRACKED (binding, ctx->ovnsb_idl) {
@@ -818,6 +866,8 @@ physical_run(struct controller_ctx *ctx, enum mf_field_id 
mff_ovn_geneve,
 const struct sbrec_multicast_group *mc;
 struct ofpbuf remote_ofpacts;
 ofpbuf_init(_ofpacts, 0);
+struct hmap new_multicast_group_uuids =
+HMAP_INITIALIZER(_multicast_group_uuids);
 SBREC_MULTICAST_GROUP_FOR_EACH (mc, ctx->ovnsb_idl) {
 /* As multicast groups are always reprocessed each time,
  * the first step is to clean the old flows for the group
@@ -825,7 +875,14 @@ physical_run(struct controller_ctx *ctx, enum mf_field_id 
mff_ovn_geneve,
 ofctrl_remove_flows(>header_.uuid);
 consider_mc_group(mff_ovn_geneve, ct_zones,
   local_datapaths, mc, , _ofpacts);
+struct uuid_hash_node *hash_node = xzalloc(sizeof *hash_node);
+hash_node->uuid = mc->header_.uuid;
+hmap_insert(_mul

[ovs-dev] [PATCH v4 2/2] Add wrapper scripts for *ctl commands

2016-08-14 Thread Ryan Moats 
This commit creates wrapper scripts for the *ctl commands to use
--dry-run or --read-only as appropriate, and to allow for log level
setting vi ovs-appctl without allowing full access to ovs-appctl.
The debian install files for ovn-common, openvswitch-common and
openvswitch-vswitch have been modified to include the wrapper
scripts as part of files to be installed.

Tests have been added to make sure that the wrapper scripts
don't actually do anything when asked to perform a write operation.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 debian/openvswitch-common.install |   2 +
 debian/openvswitch-switch.install |   2 +
 debian/ovn-common.install |   2 +
 ovn/utilities/automake.mk |   8 ++-
 ovn/utilities/ovn-nbread  |   2 +
 ovn/utilities/ovn-sbread  |   2 +
 tests/ovn-nbctl.at| 103 +++
 tests/ovn-sbctl.at|  46 
 tests/ovs-ofctl.at|  43 +++
 tests/ovs-vsctl.at|  90 +++
 tests/vlog.at |  27 +++
 tests/vtep-ctl.at | 145 ++
 utilities/automake.mk |  10 ++-
 utilities/ovs-appsetlog   |  40 +++
 utilities/ovs-dpread  |   2 +
 utilities/ovs-ofread  |   2 +
 utilities/ovs-vsread  |   2 +
 vtep/automake.mk  |   5 +-
 vtep/vtep-read|   2 +
 19 files changed, 530 insertions(+), 5 deletions(-)
 create mode 100755 ovn/utilities/ovn-nbread
 create mode 100755 ovn/utilities/ovn-sbread
 create mode 100755 utilities/ovs-appsetlog
 create mode 100755 utilities/ovs-dpread
 create mode 100755 utilities/ovs-ofread
 create mode 100755 utilities/ovs-vsread
 create mode 100755 vtep/vtep-read

diff --git a/debian/openvswitch-common.install 
b/debian/openvswitch-common.install
index ebb7d5c..8b0ed03 100644
--- a/debian/openvswitch-common.install
+++ b/debian/openvswitch-common.install
@@ -7,6 +7,8 @@ usr/bin/ovs-pki
 usr/bin/ovsdb-client
 usr/sbin/ovs-bugtool
 usr/share/openvswitch/bugtool-plugins
+usr/share/openvswitch/scripts/ovs-appsetlog
 usr/share/openvswitch/scripts/ovs-bugtool-*
 usr/share/openvswitch/scripts/ovs-lib
+usr/share/openvswitch/scripts/ovs-ofread
 usr/lib/lib*.so.*
diff --git a/debian/openvswitch-switch.install 
b/debian/openvswitch-switch.install
index bfb391f..934915d 100644
--- a/debian/openvswitch-switch.install
+++ b/debian/openvswitch-switch.install
@@ -12,5 +12,7 @@ usr/sbin/ovs-vswitchd
 usr/sbin/ovsdb-server
 usr/share/openvswitch/scripts/ovs-check-dead-ifs
 usr/share/openvswitch/scripts/ovs-ctl
+usr/share/openvswitch/scripts/ovs-dpread
 usr/share/openvswitch/scripts/ovs-save
+usr/share/openvswitch/scripts/ovs-vsread
 usr/share/openvswitch/vswitch.ovsschema
diff --git a/debian/ovn-common.install b/debian/ovn-common.install
index acb1dc9..b9fae25 100644
--- a/debian/ovn-common.install
+++ b/debian/ovn-common.install
@@ -1,3 +1,5 @@
 usr/bin/ovn-nbctl
 usr/bin/ovn-sbctl
 usr/share/openvswitch/scripts/ovn-ctl
+usr/share/openvswitch/scripts/ovn-nbread
+usr/share/openvswitch/scripts/ovn-sbread
diff --git a/ovn/utilities/automake.mk b/ovn/utilities/automake.mk
index d84368c..c78a07f 100644
--- a/ovn/utilities/automake.mk
+++ b/ovn/utilities/automake.mk
@@ -1,5 +1,7 @@
 scripts_SCRIPTS += \
-ovn/utilities/ovn-ctl
+ovn/utilities/ovn-ctl \
+ovn/utilities/ovn-nbread \
+ovn/utilities/ovn-sbread
 
 man_MANS += \
 ovn/utilities/ovn-ctl.8 \
@@ -18,7 +20,9 @@ EXTRA_DIST += \
 ovn/utilities/ovn-ctl.8.xml \
 ovn/utilities/ovn-docker-overlay-driver \
 ovn/utilities/ovn-docker-underlay-driver \
-ovn/utilities/ovn-nbctl.8.xml
+ovn/utilities/ovn-nbctl.8.xml \
+ovn/utilities/ovn-nbread \
+ovn/utilities/ovn-sbread
 
 DISTCLEANFILES += \
 ovn/utilities/ovn-ctl.8 \
diff --git a/ovn/utilities/ovn-nbread b/ovn/utilities/ovn-nbread
new file mode 100755
index 000..27c9b71
--- /dev/null
+++ b/ovn/utilities/ovn-nbread
@@ -0,0 +1,2 @@
+#! /bin/sh
+exec ovn-nbctl --dry-run "$@"
diff --git a/ovn/utilities/ovn-sbread b/ovn/utilities/ovn-sbread
new file mode 100755
index 000..d5c3f44
--- /dev/null
+++ b/ovn/utilities/ovn-sbread
@@ -0,0 +1,2 @@
+#! /bin/sh
+exec ovn-sbctl --dry-run "$@"
diff --git a/tests/ovn-nbctl.at b/tests/ovn-nbctl.at
index 5357ced..615e0fc 100644
--- a/tests/ovn-nbctl.at
+++ b/tests/ovn-nbctl.at
@@ -485,3 +485,106 @@ IPv6 Routes
 
 OVN_NBCTL_TEST_STOP
 AT_CLEANUP
+
+dnl -
+
+AT_SETUP([ovn-nbread - negative tests])
+OVN_NBCTL_TEST_START
+ovn-nbctl ls-add base
+ovn-nbctl ls-list > expout
+ovn-nbread init
+AT_CHECK([ovn-nbread ls-list], [0], [expout])
+ovn-nbread ls-add canary
+AT_CHECK([ovn-nbread ls-list], [0], [expout])
+ovn-nbread ls-del base
+AT_CHECK([ovn-nbread ls-list], [0], [expout])
+
+ovn-nbread acl-add base to-lpo

[ovs-dev] [PATCH v4 1/2] Add read-only option to ovs-dpctl and ovs-ofctl commands.

2016-08-14 Thread Ryan Moats 
ovs-dpctl and ovs-ofctl lack a read-only option to prevent
running of commands that perform read-write operations.  Add
it and the necessary scaffolding to each.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---

v1->v2:
 Fixed typo in usage string

 lib/command-line.c |  50 +
 lib/command-line.h |   6 +-
 lib/db-ctl-base.h  |   2 +-
 lib/dpctl.c|  44 ++--
 lib/dpctl.h|   3 +
 ovsdb/ovsdb-tool.c |  28 
 tests/ovstest.c|   4 +-
 tests/test-bitmap.c|   6 +-
 tests/test-ccmap.c |   6 +-
 tests/test-classifier.c|  28 
 tests/test-cmap.c  |   6 +-
 tests/test-conntrack.c |   6 +-
 tests/test-heap.c  |  14 ++--
 tests/test-jsonrpc.c   |  10 +--
 tests/test-netlink-conntrack.c |   8 +--
 tests/test-ovn.c   |  24 +++
 tests/test-ovsdb.c |  78 ++--
 tests/test-reconnect.c |  32 -
 tests/test-util.c  |  34 -
 tests/test-vconn.c |  18 ++---
 utilities/ovs-dpctl.c  |   7 ++
 utilities/ovs-ofctl.8.in   |   3 +
 utilities/ovs-ofctl.c  | 160 ++---
 23 files changed, 317 insertions(+), 260 deletions(-)

diff --git a/lib/command-line.c b/lib/command-line.c
index bda5ed6..7f30646 100644
--- a/lib/command-line.c
+++ b/lib/command-line.c
@@ -87,20 +87,11 @@ ovs_cmdl_print_options(const struct option options[])
 ds_destroy();
 }

-/* Runs the command designated by argv[0] within the command table specified by
- * 'commands', which must be terminated by a command whose 'name' member is a
- * null pointer.
- *
- * Command-line options should be stripped off, so that a typical invocation
- * looks like:
- *struct ovs_cmdl_context ctx = {
- *.argc = argc - optind,
- *.argv = argv + optind,
- *};
- *ovs_cmdl_run_command(, my_commands);
- * */
-void
-ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const struct 
ovs_cmdl_command commands[])
+/* Whether to commit changes or not. */
+static bool dry_run;
+
+static void
+_ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const struct 
ovs_cmdl_command commands[])
 {
 const struct ovs_cmdl_command *p;

@@ -118,6 +109,9 @@ ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const 
struct ovs_cmdl_command
 VLOG_FATAL("'%s' command takes at most %d arguments",
p->name, p->max_args);
 } else {
+if (p->mode == OVS_RW && dry_run) {
+return;
+}
 p->handler(ctx);
 if (ferror(stdout)) {
 VLOG_FATAL("write to stdout failed");
@@ -132,6 +126,34 @@ ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const 
struct ovs_cmdl_command

 VLOG_FATAL("unknown command '%s'; use --help for help", ctx->argv[0]);
 }
+
+/* Runs the command designated by argv[0] within the command table specified by
+ * 'commands', which must be terminated by a command whose 'name' member is a
+ * null pointer.
+ *
+ * Command-line options should be stripped off, so that a typical invocation
+ * looks like:
+ *struct ovs_cmdl_context ctx = {
+ *.argc = argc - optind,
+ *.argv = argv + optind,
+ *};
+ *ovs_cmdl_run_command(, my_commands);
+ * */
+void
+ovs_cmdl_run_command(struct ovs_cmdl_context *ctx,
+ const struct ovs_cmdl_command commands[])
+{
+dry_run = false;
+_ovs_cmdl_run_command(ctx, commands);
+}
+
+void
+ovs_cmdl_run_command_dry_run(struct ovs_cmdl_context *ctx,
+ const struct ovs_cmdl_command commands[])
+{
+dry_run = true;
+_ovs_cmdl_run_command(ctx, commands);
+}
 
 /* Process title. */

diff --git a/lib/command-line.h b/lib/command-line.h
index e9e3b7b..d0c4c3e 100644
--- a/lib/command-line.h
+++ b/lib/command-line.h
@@ -41,12 +41,16 @@ struct ovs_cmdl_command {
 int min_args;
 int max_args;
 ovs_cmdl_handler handler;
+enum { OVS_RO, OVS_RW } mode;/* Does this command modify things? */
 };

 char *ovs_cmdl_long_options_to_short_options(const struct option *options);
 void ovs_cmdl_print_options(const struct option *options);
 void ovs_cmdl_print_commands(const struct ovs_cmdl_command *commands);
-void ovs_cmdl_run_command(struct ovs_cmdl_context *, const struct 
ovs_cmdl_command[]);
+void ovs_cmdl_run_command(struct ovs_cmdl_context *,
+  const struct ovs_cmdl_command[]);
+void ovs_cmdl_run_command_dry_run(struct ovs_cmdl_context *,
+  const struct ovs_cmdl_command[]);

 void ovs_cmdl_proctitle_init(int argc, char **argv);
 #if defined(__FreeBSD__) || defined(__NetBSD__)
diff --git a/lib/db-ctl-base.h b/lib/db-ctl-base.h
index 0f

[ovs-dev] [PATCH v4 0/2] Read only versions of the *ctl commands

2016-08-14 Thread Ryan Moats 
This patch series adds read-only wrapper scripts around the various
*ctl commands that leverage the --dry-run command line option for
scripts that modify a database.

For commands that don't have the --dry-run option (ovs-dpctl
and ovs-ofctl) a new --read-only option is added to prevent commands
tagged as read/write from being executed.

Negative tests are included to verify that the wrapper scripts
don't change things.

v3->v4:
  rebased

v2->v3:
  Changed from --dry-run to --read-only for ovs-dpctl and ovs-ofctl
  
v1->v2:
  Fixed typo in patch set one.
  Added unit test for ovs-appsetlog
  Added new scripts into approriate debian package definitions.

Ryan Moats (2):
  Add read-only option to ovs-dpctl and ovs-ofctl commands.
  Add wrapper scripts for *ctl commands

 debian/openvswitch-common.install |   2 +
 debian/openvswitch-switch.install |   2 +
 debian/ovn-common.install |   2 +
 lib/command-line.c|  51 
 lib/command-line.h|   6 +-
 lib/db-ctl-base.h |   2 +-
 lib/dpctl.c   |  47 ++-
 lib/dpctl.h   |   3 +
 ovn/utilities/automake.mk |   8 +-
 ovn/utilities/ovn-nbread  |   2 +
 ovn/utilities/ovn-sbread  |   2 +
 ovsdb/ovsdb-tool.c|  28 +++
 tests/ovn-nbctl.at| 103 
 tests/ovn-sbctl.at|  46 +++
 tests/ovs-ofctl.at|  43 ++
 tests/ovs-vsctl.at|  90 +
 tests/ovstest.c   |   4 +-
 tests/test-bitmap.c   |   6 +-
 tests/test-ccmap.c|   6 +-
 tests/test-classifier.c   |  28 +++
 tests/test-cmap.c |   6 +-
 tests/test-conntrack.c|   6 +-
 tests/test-heap.c |  14 ++--
 tests/test-jsonrpc.c  |  10 +--
 tests/test-netlink-conntrack.c|   8 +-
 tests/test-ovn.c  |  26 +++
 tests/test-ovsdb.c|  80 +--
 tests/test-reconnect.c|  32 
 tests/test-util.c |  34 
 tests/test-vconn.c|  18 ++---
 tests/vlog.at |  27 +++
 tests/vtep-ctl.at | 145 ++
 utilities/automake.mk |  10 ++-
 utilities/ovs-appsetlog   |  40 ++
 utilities/ovs-dpctl.c |   7 ++
 utilities/ovs-dpread  |   2 +
 utilities/ovs-ofctl.8.in  |   3 +
 utilities/ovs-ofctl.c | 160 +-
 utilities/ovs-ofread  |   2 +
 utilities/ovs-vsread  |   2 +
 vtep/automake.mk  |   5 +-
 vtep/vtep-read|   2 +
 42 files changed, 853 insertions(+), 267 deletions(-)
 create mode 100755 ovn/utilities/ovn-nbread
 create mode 100755 ovn/utilities/ovn-sbread
 create mode 100755 utilities/ovs-appsetlog
 create mode 100755 utilities/ovs-dpread
 create mode 100755 utilities/ovs-ofread
 create mode 100755 utilities/ovs-vsread
 create mode 100755 vtep/vtep-read

-- 
2.7.4

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [ovs-dev, v3, 1/2] Add read-only option to ovs-dpctl and ovs-ofctl commands.

2016-08-14 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 08/14/2016 10:22:33 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/14/2016 10:22 PM
> Subject: Re: [ovs-dev, v3, 1/2] Add read-only option to ovs-dpctl
> and ovs-ofctl commands.
>
> On Mon, Aug 15, 2016 at 12:17:34AM +, Ryan Moats wrote:
> > ovs-dpctl and ovs-ofctl lack a read-only option to prevent
> > running of commands that perform read-write operations.  Add
> > it and the necessary scaffolding to each.
> >
> > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> > ---
> >
> > v1->v2:
> >  Fixed typo in usage string
>
> I'm confused here, almost every patch fails for me:

I admit to also being confused - I needed a rebase on
tests/test-ovsdb.c (which is why v4 is coming out),
but I double-checked and my master is up to date...
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH] Simplify partial map Py3 IDL test added by commit a59912a0

2016-08-14 Thread Ryan Moats 
Commit a59912a0 ("python: Add support for partial map
and partial set updates") added unit tests for the partial
map function for the python IDL.  However, because Python3
doesn't order dictionaries consistently, this
test is a crap shoot for systems that support Python3.

As a short term fix, do not use a dictionary with multiple
elements for the partial map test case.

Change-Id: Ibdec10ebd895051321b9bff7d9fe8a7e0bd9eb88
Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 tests/ovsdb-idl.at  | 8 
 tests/test-ovsdb.py | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/tests/ovsdb-idl.at b/tests/ovsdb-idl.at
index c61d2e7..b2899b6 100644
--- a/tests/ovsdb-idl.at
+++ b/tests/ovsdb-idl.at
@@ -1110,14 +1110,14 @@ OVSDB_CHECK_IDL_PARTIAL_UPDATE_MAP_COLUMN([map, simple2 
idl-partial-update-map-c
 
 OVSDB_CHECK_IDL_PY([partial-map idl],
 [['["idltest", {"op":"insert", "table":"simple2",
-
"row":{"name":"myString1","smap":["map",[["key1","value1"],["key2","value2"]]]} 
}]']
+"row":{"name":"myString1","smap":["map",[["key1","value1"]]]} 
}]']
 ],
   [?simple2:name,smap,imap 'partialmapinsertelement' 'partialmapdelelement'],
-[[000: name=myString1 smap={key2: value2 key1: value1} imap={}
+[[000: name=myString1 smap={key1: value1} imap={}
 001: commit, status=success
-002: name=String2 smap={key2: value2 key1: myList1} imap={3: myids2}
+002: name=String2 smap={key1: myList1} imap={3: myids2}
 003: commit, status=success
-004: name=String2 smap={key1: myList1} imap={3: myids2}
+004: name=String2 smap={} imap={3: myids2}
 005: done
 ]])
 
diff --git a/tests/test-ovsdb.py b/tests/test-ovsdb.py
index 86f0168..2ea2bd5 100644
--- a/tests/test-ovsdb.py
+++ b/tests/test-ovsdb.py
@@ -450,7 +450,7 @@ def idl_set(idl, commands, step):
 row.__setattr__('name', 'String2')
 elif name == 'partialmapdelelement':
 row = idltest_find_simple2(idl, 'String2')
-row.delkey('smap', 'key2', 'value2')
+row.delkey('smap', 'key1', 'myList1')
 elif name == 'partialrenamesetadd':
 row = idltest_find_simple3(idl, 'mySet1')
 row.addvalue('uset',
-- 
2.7.4 (Apple Git-66)

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] ovs-ctl: Properly handle shell quoting inos-release.

2016-08-14 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 08/14/2016 07:38:08 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org, Matthew Mulsow/Austin/IBM@IBMUS
> Date: 08/14/2016 07:38 PM
> Subject: Re: [ovs-dev] [PATCH] ovs-ctl: Properly handle shell
> quoting inos-release.
>
> On Sun, Aug 14, 2016 at 06:57:54PM -0500, Ryan Moats wrote:
> > "dev" <dev-boun...@openvswitch.org> wrote on 08/14/2016 02:39:56 PM:
> >
> > > From: Ben Pfaff <b...@ovn.org>
> > > To: dev@openvswitch.org
> > > Cc: Matthew Mulsow/Austin/IBM@IBMUS, Ben Pfaff <b...@ovn.org>
> > > Date: 08/14/2016 02:40 PM
> > > Subject: [ovs-dev] [PATCH] ovs-ctl: Properly handle shell quoting in
> > > os-release.
> > > Sent by: "dev" <dev-boun...@openvswitch.org>
> > >
> > > Until now, this code did not strip "" or '' from variable assignments
in
> > > os-release.  This fixes the problem.
> > >
> > > CC: Matt Mulsow <mamul...@us.ibm.com>
> > > Fixes: c60d6b096436 ("ovs-ctl: support populating system info from /
> > > etc/os-release")
> > > Signed-off-by: Ben Pfaff <b...@ovn.org>
> > > ---
> > > I have not tested this.
> > >
> > >  utilities/ovs-ctl.in | 4 ++--
> > >  1 file changed, 2 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/utilities/ovs-ctl.in b/utilities/ovs-ctl.in
> > > index ff3eaf2..dc275f8 100755
> > > --- a/utilities/ovs-ctl.in
> > > +++ b/utilities/ovs-ctl.in
> > > @@ -523,8 +523,8 @@ set_defaults () {
> > >  SYSTEM_TYPE=`cat $type_file`
> > >  SYSTEM_VERSION=`cat $version_file`
> > >  elif test -e "@sysconfdir@/os-release"; then
> > > -SYSTEM_TYPE=`awk -F= '/^ID=/{print $2}'
@sysconfdir@/os-release`
> > > -SYSTEM_VERSION=`awk -F= '/^VERSION_ID=/{print $2}'
> > > @sysconfdir@/os-release`
> > > +SYSTEM_TYPE=`. '@sysconfdir@/os-release' && echo "$ID"`
> > > +SYSTEM_VERSION=`. '@sysconfdir@/os-release' && echo
> > "$VERSION_ID"`
> > >  elif (lsb_release --id) >/dev/null 2>&1; then
> > >  SYSTEM_TYPE=`lsb_release --id -s`
> > >  system_release=`lsb_release --release -s`
> > > --
> > > 2.1.3
> >
> > I've not seen this problem in our tests, but I have seen in it our
wild,
> > so...
> >
> > Acked-by: Ryan Moats <rmo...@us.ibm.com>
>
> I should have included:
>
> Requested-by: Matt Mulsow <mamul...@us.ibm.com>
> Requested-at: https://github.com/openvswitch/ovs/pull/148
>
> Anyway, I added those and applied this, thanks for the review.
>

Thanks, this hit us at the start of the weekend and folks have been
dealing with it...

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v3 2/2] Add wrapper scripts for *ctl commands

2016-08-14 Thread Ryan Moats 
This commit creates wrapper scripts for the *ctl commands to use
--dry-run or --read-only as appropriate, and to allow for log level
setting vi ovs-appctl without allowing full access to ovs-appctl.
The debian install files for ovn-common, openvswitch-common and
openvswitch-vswitch have been modified to include the wrapper
scripts as part of files to be installed.

Tests have been added to make sure that the wrapper scripts
don't actually do anything when asked to perform a write operation.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 debian/openvswitch-common.install |   2 +
 debian/openvswitch-switch.install |   2 +
 debian/ovn-common.install |   2 +
 ovn/utilities/automake.mk |   8 ++-
 ovn/utilities/ovn-nbread  |   2 +
 ovn/utilities/ovn-sbread  |   2 +
 tests/ovn-nbctl.at| 103 +++
 tests/ovn-sbctl.at|  46 
 tests/ovs-ofctl.at|  43 +++
 tests/ovs-vsctl.at|  90 +++
 tests/vlog.at |  27 +++
 tests/vtep-ctl.at | 145 ++
 utilities/automake.mk |  10 ++-
 utilities/ovs-appsetlog   |  40 +++
 utilities/ovs-dpread  |   2 +
 utilities/ovs-ofread  |   2 +
 utilities/ovs-vsread  |   2 +
 vtep/automake.mk  |   5 +-
 vtep/vtep-read|   2 +
 19 files changed, 530 insertions(+), 5 deletions(-)
 create mode 100755 ovn/utilities/ovn-nbread
 create mode 100755 ovn/utilities/ovn-sbread
 create mode 100755 utilities/ovs-appsetlog
 create mode 100755 utilities/ovs-dpread
 create mode 100755 utilities/ovs-ofread
 create mode 100755 utilities/ovs-vsread
 create mode 100755 vtep/vtep-read

diff --git a/debian/openvswitch-common.install 
b/debian/openvswitch-common.install
index ebb7d5c..8b0ed03 100644
--- a/debian/openvswitch-common.install
+++ b/debian/openvswitch-common.install
@@ -7,6 +7,8 @@ usr/bin/ovs-pki
 usr/bin/ovsdb-client
 usr/sbin/ovs-bugtool
 usr/share/openvswitch/bugtool-plugins
+usr/share/openvswitch/scripts/ovs-appsetlog
 usr/share/openvswitch/scripts/ovs-bugtool-*
 usr/share/openvswitch/scripts/ovs-lib
+usr/share/openvswitch/scripts/ovs-ofread
 usr/lib/lib*.so.*
diff --git a/debian/openvswitch-switch.install 
b/debian/openvswitch-switch.install
index bfb391f..934915d 100644
--- a/debian/openvswitch-switch.install
+++ b/debian/openvswitch-switch.install
@@ -12,5 +12,7 @@ usr/sbin/ovs-vswitchd
 usr/sbin/ovsdb-server
 usr/share/openvswitch/scripts/ovs-check-dead-ifs
 usr/share/openvswitch/scripts/ovs-ctl
+usr/share/openvswitch/scripts/ovs-dpread
 usr/share/openvswitch/scripts/ovs-save
+usr/share/openvswitch/scripts/ovs-vsread
 usr/share/openvswitch/vswitch.ovsschema
diff --git a/debian/ovn-common.install b/debian/ovn-common.install
index acb1dc9..b9fae25 100644
--- a/debian/ovn-common.install
+++ b/debian/ovn-common.install
@@ -1,3 +1,5 @@
 usr/bin/ovn-nbctl
 usr/bin/ovn-sbctl
 usr/share/openvswitch/scripts/ovn-ctl
+usr/share/openvswitch/scripts/ovn-nbread
+usr/share/openvswitch/scripts/ovn-sbread
diff --git a/ovn/utilities/automake.mk b/ovn/utilities/automake.mk
index d84368c..c78a07f 100644
--- a/ovn/utilities/automake.mk
+++ b/ovn/utilities/automake.mk
@@ -1,5 +1,7 @@
 scripts_SCRIPTS += \
-ovn/utilities/ovn-ctl
+ovn/utilities/ovn-ctl \
+ovn/utilities/ovn-nbread \
+ovn/utilities/ovn-sbread
 
 man_MANS += \
 ovn/utilities/ovn-ctl.8 \
@@ -18,7 +20,9 @@ EXTRA_DIST += \
 ovn/utilities/ovn-ctl.8.xml \
 ovn/utilities/ovn-docker-overlay-driver \
 ovn/utilities/ovn-docker-underlay-driver \
-ovn/utilities/ovn-nbctl.8.xml
+ovn/utilities/ovn-nbctl.8.xml \
+ovn/utilities/ovn-nbread \
+ovn/utilities/ovn-sbread
 
 DISTCLEANFILES += \
 ovn/utilities/ovn-ctl.8 \
diff --git a/ovn/utilities/ovn-nbread b/ovn/utilities/ovn-nbread
new file mode 100755
index 000..27c9b71
--- /dev/null
+++ b/ovn/utilities/ovn-nbread
@@ -0,0 +1,2 @@
+#! /bin/sh
+exec ovn-nbctl --dry-run "$@"
diff --git a/ovn/utilities/ovn-sbread b/ovn/utilities/ovn-sbread
new file mode 100755
index 000..d5c3f44
--- /dev/null
+++ b/ovn/utilities/ovn-sbread
@@ -0,0 +1,2 @@
+#! /bin/sh
+exec ovn-sbctl --dry-run "$@"
diff --git a/tests/ovn-nbctl.at b/tests/ovn-nbctl.at
index 5357ced..615e0fc 100644
--- a/tests/ovn-nbctl.at
+++ b/tests/ovn-nbctl.at
@@ -485,3 +485,106 @@ IPv6 Routes
 
 OVN_NBCTL_TEST_STOP
 AT_CLEANUP
+
+dnl -
+
+AT_SETUP([ovn-nbread - negative tests])
+OVN_NBCTL_TEST_START
+ovn-nbctl ls-add base
+ovn-nbctl ls-list > expout
+ovn-nbread init
+AT_CHECK([ovn-nbread ls-list], [0], [expout])
+ovn-nbread ls-add canary
+AT_CHECK([ovn-nbread ls-list], [0], [expout])
+ovn-nbread ls-del base
+AT_CHECK([ovn-nbread ls-list], [0], [expout])
+
+ovn-nbread acl-add base to-lpo

[ovs-dev] [PATCH v3 1/2] Add read-only option to ovs-dpctl and ovs-ofctl commands.

2016-08-14 Thread Ryan Moats 
ovs-dpctl and ovs-ofctl lack a read-only option to prevent
running of commands that perform read-write operations.  Add
it and the necessary scaffolding to each.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---

v1->v2:
 Fixed typo in usage string

 lib/command-line.c |  50 +
 lib/command-line.h |   6 +-
 lib/db-ctl-base.h  |   2 +-
 lib/dpctl.c|  44 ++--
 lib/dpctl.h|   3 +
 ovsdb/ovsdb-tool.c |  28 
 tests/ovstest.c|   4 +-
 tests/test-bitmap.c|   6 +-
 tests/test-ccmap.c |   6 +-
 tests/test-classifier.c|  28 
 tests/test-cmap.c  |   6 +-
 tests/test-conntrack.c |   6 +-
 tests/test-heap.c  |  14 ++--
 tests/test-jsonrpc.c   |  10 +--
 tests/test-netlink-conntrack.c |   8 +--
 tests/test-ovn.c   |  24 +++
 tests/test-ovsdb.c |  78 ++--
 tests/test-reconnect.c |  32 -
 tests/test-util.c  |  34 -
 tests/test-vconn.c |  18 ++---
 utilities/ovs-dpctl.c  |   7 ++
 utilities/ovs-ofctl.8.in   |   3 +
 utilities/ovs-ofctl.c  | 160 ++---
 23 files changed, 317 insertions(+), 260 deletions(-)

diff --git a/lib/command-line.c b/lib/command-line.c
index bda5ed6..7f30646 100644
--- a/lib/command-line.c
+++ b/lib/command-line.c
@@ -87,20 +87,11 @@ ovs_cmdl_print_options(const struct option options[])
 ds_destroy();
 }

-/* Runs the command designated by argv[0] within the command table specified by
- * 'commands', which must be terminated by a command whose 'name' member is a
- * null pointer.
- *
- * Command-line options should be stripped off, so that a typical invocation
- * looks like:
- *struct ovs_cmdl_context ctx = {
- *.argc = argc - optind,
- *.argv = argv + optind,
- *};
- *ovs_cmdl_run_command(, my_commands);
- * */
-void
-ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const struct 
ovs_cmdl_command commands[])
+/* Whether to commit changes or not. */
+static bool dry_run;
+
+static void
+_ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const struct 
ovs_cmdl_command commands[])
 {
 const struct ovs_cmdl_command *p;

@@ -118,6 +109,9 @@ ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const 
struct ovs_cmdl_command
 VLOG_FATAL("'%s' command takes at most %d arguments",
p->name, p->max_args);
 } else {
+if (p->mode == OVS_RW && dry_run) {
+return;
+}
 p->handler(ctx);
 if (ferror(stdout)) {
 VLOG_FATAL("write to stdout failed");
@@ -132,6 +126,34 @@ ovs_cmdl_run_command(struct ovs_cmdl_context *ctx, const 
struct ovs_cmdl_command

 VLOG_FATAL("unknown command '%s'; use --help for help", ctx->argv[0]);
 }
+
+/* Runs the command designated by argv[0] within the command table specified by
+ * 'commands', which must be terminated by a command whose 'name' member is a
+ * null pointer.
+ *
+ * Command-line options should be stripped off, so that a typical invocation
+ * looks like:
+ *struct ovs_cmdl_context ctx = {
+ *.argc = argc - optind,
+ *.argv = argv + optind,
+ *};
+ *ovs_cmdl_run_command(, my_commands);
+ * */
+void
+ovs_cmdl_run_command(struct ovs_cmdl_context *ctx,
+ const struct ovs_cmdl_command commands[])
+{
+dry_run = false;
+_ovs_cmdl_run_command(ctx, commands);
+}
+
+void
+ovs_cmdl_run_command_dry_run(struct ovs_cmdl_context *ctx,
+ const struct ovs_cmdl_command commands[])
+{
+dry_run = true;
+_ovs_cmdl_run_command(ctx, commands);
+}
 
 /* Process title. */

diff --git a/lib/command-line.h b/lib/command-line.h
index e9e3b7b..d0c4c3e 100644
--- a/lib/command-line.h
+++ b/lib/command-line.h
@@ -41,12 +41,16 @@ struct ovs_cmdl_command {
 int min_args;
 int max_args;
 ovs_cmdl_handler handler;
+enum { OVS_RO, OVS_RW } mode;/* Does this command modify things? */
 };

 char *ovs_cmdl_long_options_to_short_options(const struct option *options);
 void ovs_cmdl_print_options(const struct option *options);
 void ovs_cmdl_print_commands(const struct ovs_cmdl_command *commands);
-void ovs_cmdl_run_command(struct ovs_cmdl_context *, const struct 
ovs_cmdl_command[]);
+void ovs_cmdl_run_command(struct ovs_cmdl_context *,
+  const struct ovs_cmdl_command[]);
+void ovs_cmdl_run_command_dry_run(struct ovs_cmdl_context *,
+  const struct ovs_cmdl_command[]);

 void ovs_cmdl_proctitle_init(int argc, char **argv);
 #if defined(__FreeBSD__) || defined(__NetBSD__)
diff --git a/lib/db-ctl-base.h b/lib/db-ctl-base.h
index 0f

[ovs-dev] [PATCH v3 0/2] Read only versions of the *ctl commands

2016-08-14 Thread Ryan Moats 
This patch series adds read-only wrapper scripts around the various
*ctl commands that leverage the --dry-run command line option for
scripts that modify a database.

For commands that don't have the --dry-run option (ovs-dpctl
and ovs-ofctl) a new --read-only option is added to prevent commands
tagged as read/write from being executed.

Negative tests are included to verify that the wrapper scripts
don't change things.

v2->v3:
  Changed from --dry-run to --read-only for ovs-dpctl and ovs-ofctl
  
v1->v2:
  Fixed typo in patch set one.
  Added unit test for ovs-appsetlog
  Added new scripts into approriate debian package definitions.

Ryan Moats (2):
  Add read-only option to ovs-dpctl and ovs-ofctl commands.
  Add wrapper scripts for *ctl commands

 debian/openvswitch-common.install |   2 +
 debian/openvswitch-switch.install |   2 +
 debian/ovn-common.install |   2 +
 lib/command-line.c|  51 
 lib/command-line.h|   6 +-
 lib/db-ctl-base.h |   2 +-
 lib/dpctl.c   |  47 ++-
 lib/dpctl.h   |   3 +
 ovn/utilities/automake.mk |   8 +-
 ovn/utilities/ovn-nbread  |   2 +
 ovn/utilities/ovn-sbread  |   2 +
 ovsdb/ovsdb-tool.c|  28 +++
 tests/ovn-nbctl.at| 103 
 tests/ovn-sbctl.at|  46 +++
 tests/ovs-ofctl.at|  43 ++
 tests/ovs-vsctl.at|  90 +
 tests/ovstest.c   |   4 +-
 tests/test-bitmap.c   |   6 +-
 tests/test-ccmap.c|   6 +-
 tests/test-classifier.c   |  28 +++
 tests/test-cmap.c |   6 +-
 tests/test-conntrack.c|   6 +-
 tests/test-heap.c |  14 ++--
 tests/test-jsonrpc.c  |  10 +--
 tests/test-netlink-conntrack.c|   8 +-
 tests/test-ovn.c  |  26 +++
 tests/test-ovsdb.c|  78 +--
 tests/test-reconnect.c|  32 
 tests/test-util.c |  34 
 tests/test-vconn.c|  18 ++---
 tests/vlog.at |  27 +++
 tests/vtep-ctl.at | 145 ++
 utilities/automake.mk |  10 ++-
 utilities/ovs-appsetlog   |  40 ++
 utilities/ovs-dpctl.c |   7 ++
 utilities/ovs-dpread  |   2 +
 utilities/ovs-ofctl.8.in  |   3 +
 utilities/ovs-ofctl.c | 160 +-
 utilities/ovs-ofread  |   2 +
 utilities/ovs-vsread  |   2 +
 vtep/automake.mk  |   5 +-
 vtep/vtep-read|   2 +
 42 files changed, 852 insertions(+), 266 deletions(-)
 create mode 100755 ovn/utilities/ovn-nbread
 create mode 100755 ovn/utilities/ovn-sbread
 create mode 100755 utilities/ovs-appsetlog
 create mode 100755 utilities/ovs-dpread
 create mode 100755 utilities/ovs-ofread
 create mode 100755 utilities/ovs-vsread
 create mode 100755 vtep/vtep-read

-- 
2.7.4

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH v2 1/2] Add dry-run option to ovs-dpctl and ovs-ofctl commands.

2016-08-14 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 08/14/2016 12:36:31 AM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/14/2016 12:36 AM
> Subject: Re: [ovs-dev] [PATCH v2 1/2] Add dry-run option to ovs-
> dpctl and ovs-ofctl commands.
>
> On Thu, Aug 04, 2016 at 05:55:34PM +, Ryan Moats wrote:
> > ovs-dpctl and ovs-ofctl lack a dry-run option.  Add it
> > and the necessary scaffolding to each.
> >
> > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
>
> The point behind a --dry-run option is supposed to be that it does
> everything up to but not actually including modifying state.  For
> example, in ovs-vsctl it actually connects to the database and does the
> whole operation up to the point where it sends the transaction to the
> remote database, at which point it just doesn't do it.  This is valuable
> for the user because it means that all of the syntax gets checked and
> most of the semantics (e.g. "ovs-vsctl --dry-run del-br br0" will check
> that a bridge br0 exists).
>
> As implemented by this patch, though, --dry-run does nothing at all
> beyond verifying that there is a plausible number of parameters to a
> command that might modify state.  Then it exits silently and
> successfully, giving the user the impression that something happened.
>
> So, I think that such an option should be named something different, say
> --read-only, and should cause the utility to exit with an error message.
>
> I think that's a pretty simple change to the series so perhaps we can
> still get this in on Monday?
>

Working on it right now...

On a side note - I'm starting to see consistent failures in the bfd - decay
case...
Is this just me, or are others also seeing it?

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] ovs-ctl: Properly handle shell quoting inos-release.

2016-08-14 Thread Ryan Moats 
"dev" <dev-boun...@openvswitch.org> wrote on 08/14/2016 02:39:56 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: dev@openvswitch.org
> Cc: Matthew Mulsow/Austin/IBM@IBMUS, Ben Pfaff <b...@ovn.org>
> Date: 08/14/2016 02:40 PM
> Subject: [ovs-dev] [PATCH] ovs-ctl: Properly handle shell quoting in
> os-release.
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> Until now, this code did not strip "" or '' from variable assignments in
> os-release.  This fixes the problem.
>
> CC: Matt Mulsow <mamul...@us.ibm.com>
> Fixes: c60d6b096436 ("ovs-ctl: support populating system info from /
> etc/os-release")
> Signed-off-by: Ben Pfaff <b...@ovn.org>
> ---
> I have not tested this.
>
>  utilities/ovs-ctl.in | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/utilities/ovs-ctl.in b/utilities/ovs-ctl.in
> index ff3eaf2..dc275f8 100755
> --- a/utilities/ovs-ctl.in
> +++ b/utilities/ovs-ctl.in
> @@ -523,8 +523,8 @@ set_defaults () {
>  SYSTEM_TYPE=`cat $type_file`
>  SYSTEM_VERSION=`cat $version_file`
>  elif test -e "@sysconfdir@/os-release"; then
> -SYSTEM_TYPE=`awk -F= '/^ID=/{print $2}' @sysconfdir@/os-release`
> -SYSTEM_VERSION=`awk -F= '/^VERSION_ID=/{print $2}'
> @sysconfdir@/os-release`
> +SYSTEM_TYPE=`. '@sysconfdir@/os-release' && echo "$ID"`
> +SYSTEM_VERSION=`. '@sysconfdir@/os-release' && echo
"$VERSION_ID"`
>  elif (lsb_release --id) >/dev/null 2>&1; then
>  SYSTEM_TYPE=`lsb_release --id -s`
>  system_release=`lsb_release --release -s`
> --
> 2.1.3

I've not seen this problem in our tests, but I have seen in it our wild,
so...

Acked-by: Ryan Moats <rmo...@us.ibm.com>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH v5 1/2] ovsdb: Add/use partial set updates.

2016-08-14 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 08/14/2016 06:33:45 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: d...@openvswitch.com
> Date: 08/14/2016 06:33 PM
> Subject: Re: [ovs-dev] [PATCH v5 1/2] ovsdb: Add/use partial set updates.
>
> On Sat, Aug 06, 2016 at 05:46:29PM -0500, Ryan Moats wrote:
> > This patchset mimics the changes introduced in
> >
> >   f199df26 (ovsdb-idl: Add partial map updates functionality.)
> >   010fe7ae (ovsdb-idlc.in: Autogenerate partial map updates functions.)
> >   7251075c (tests: Add test for partial map updates.)
> >
> > but for columns that store sets of values rather than key-value
> > pairs.  These columns will now be able to use the OVSDB mutate
> > operation to transmit deltas on the wire rather than use
> > verify/update and transmit wait/update operations on the wire.
> >
> > Side effect of modifying the comments in the partial map update
> > tests.
> >
> > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
>
> This missed out on some of the fixes in:
>
> b1048e6a (ovsdb-idl: Fix issues detected in Partial Map Update
feature)
>
> so I folded in the following also, and applied this to master.
>
> Most of the code in ovsdb-set-op.[ch] is in fact copied straight from
> ovsdb-map-op.[ch], but the copyright notice was changed to say IBM
> instead of HPE.  It's appropriate to *add* a copyright notice when you
> copy code and modify it, but the existing notices should be maintained.
> I don't like to mess with copyright notices without permission, so I
> left them as-is, but I suggest that you should submit a patch to add in
> the HPE notices.
>

Yeah, I realized just as I saw your email that I had forgotten to do that,
so I just pushed a patch to address that problem...
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH] Fix copyright statements from commit f1ab6e06

2016-08-14 Thread Ryan Moats 
Commit f1ab6e06 ("Add/user partial set updates.) incorrectly
did not include HPE attribution for derived files
lib/ovsdb-set-op.[ch].  Add the attribution to correct this.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
 lib/ovsdb-set-op.c | 3 ++-
 lib/ovsdb-set-op.h | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/ovsdb-set-op.c b/lib/ovsdb-set-op.c
index f41034c..62c4621 100644
--- a/lib/ovsdb-set-op.c
+++ b/lib/ovsdb-set-op.c
@@ -1,4 +1,5 @@
-/* Copyright (C) 2016, IBM
+/* Copyright (C) 2016 Hewlett Packard Enterprise Development LP
+ * Copyright (C) 2016, IBM
  * All Rights Reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License"); you may
diff --git a/lib/ovsdb-set-op.h b/lib/ovsdb-set-op.h
index 966de28..802cd7c 100644
--- a/lib/ovsdb-set-op.h
+++ b/lib/ovsdb-set-op.h
@@ -1,4 +1,5 @@
-/* Copyright (C) 2016, IBM
+/* Copyright (C) 2016 Hewlett Packard Enterprise Development LP
+ * Copyright (C) 2016, IBM
  * All Rights Reserved.
  *
  * Licensed under the Apache License, Version 2.0 (the "License"); you may
-- 
2.7.4 (Apple Git-66)

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] ovn-controller: Remove flows created for now deleted SB database rows.

2016-08-12 Thread Ryan Moats 
Ben Pfaff <b...@ovn.org> wrote on 08/12/2016 04:48:26 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: Ryan Moats/Omaha/IBM@IBMUS
> Cc: dev@openvswitch.org
> Date: 08/12/2016 04:48 PM
> Subject: Re: [ovs-dev] [PATCH] ovn-controller: Remove flows created
> for now deleted SB database rows.
>
> On Fri, Aug 12, 2016 at 02:47:09PM -0700, Ben Pfaff wrote:
> > On Thu, Jul 28, 2016 at 05:12:04PM -0500, Ryan Moats wrote:
> > >
> > > Ben Pfaff <b...@ovn.org> wrote on 07/28/2016 04:23:57 PM:
> > >
> > > > From: Ben Pfaff <b...@ovn.org>
> > > > To: Ryan Moats/Omaha/IBM@IBMUS
> > > > Cc: dev@openvswitch.org
> > > > Date: 07/28/2016 04:24 PM
> > > > Subject: Re: [ovs-dev] [PATCH] ovn-controller: Remove flows created
> > > > for now deleted SB database rows.
> > > >
> > > > On Thu, Jul 28, 2016 at 07:54:30PM +, Ryan Moats wrote:
> > > > > Ensure that rows created for deleted port binding and
> > > > > multicast group rows are cleared when doing full processing.
> > > > >
> > > > > Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
> > > >
> > > > I'm choosing to overlook storing UUIDs as strings in a set of
strings.
> > > >
> > > > How about this simplification?
> > > >
> > > > --8<--cut here-->
8--
> > > >
> > > > diff --git a/ovn/controller/physical.c b/ovn/controller/physical.c
> > > > index 6bc0095..a4a8fcf 100644
> > > > --- a/ovn/controller/physical.c
> > > > +++ b/ovn/controller/physical.c
> > > > @@ -598,25 +598,22 @@ consider_mc_group(enum mf_field_id
mff_ovn_geneve,
> > > >  sset_destroy(_chassis);
> > > >  }
> > > >
> > > > +/* Deletes the flows whose UUIDs are in 'old' but not 'new', and
> > > > then replaces
> > > > + * 'old' by 'new'. */
> > > >  static void
> > > >  rationalize_ssets_and_delete_flows(struct sset *old, struct sset
*new)
> > > >  {
> > > > -const char *uuid_s, *next_uuid;
> > > > -SSET_FOR_EACH_SAFE (uuid_s, next_uuid, old) {
> > > > +const char *uuid_s;
> > > > +SSET_FOR_EACH (uuid_s, old) {
> > > >  if (!sset_find(new, uuid_s)) {
> > > >  struct uuid uuid;
> > > >  if (uuid_from_string(, uuid_s)) {
> > > >  ofctrl_remove_flows();
> > > >  }
> > > > -sset_find_and_delete(old, uuid_s);
> > > >  }
> > > >  }
> > > > -SSET_FOR_EACH_SAFE (uuid_s, next_uuid, new) {
> > > > -if (!sset_find(old, uuid_s)) {
> > > > -sset_add(old, uuid_s);
> > > > -}
> > > > -sset_find_and_delete(new, uuid_s);
> > > > -}
> > > > +sset_swap(old, new);
> > > > +sset_clear(new);
> > > >  }
> > > >
> > > >  void
> > > >
> > >
> > > Works for me... Ryan
> >
> > I noticed a memory leak in this patch, so I folded in the following and
> > applied this to master.
>
> Oh, whoops, there was a newer version.  I've un-applied this and I'll
> look at the newer one.
>

Yeah, I decided to take you at your word in re using ssets for string
representation of UUIDs and replaced that hack with hmaps of the
actual uuids...

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

[ovs-dev] [PATCH v2] ovn-controller: Remove flows created for now deleted SB database rows.

2016-08-12 Thread Ryan Moats 
Ensure that rows created for deleted port binding and
multicast group rows are cleared when doing full processing.

Signed-off-by: Ryan Moats <rmo...@us.ibm.com>
---
v1->v2:
  - replace use of ssets for storing UUIDs as strings with hmaps for
storing the UUID structs themselves
  - include an optimization suggested by Ben when reviewing v1

 ovn/controller/physical.c | 56 ++-
 1 file changed, 55 insertions(+), 1 deletion(-)

diff --git a/ovn/controller/physical.c b/ovn/controller/physical.c
index 4448756..a27c1ac 100644
--- a/ovn/controller/physical.c
+++ b/ovn/controller/physical.c
@@ -20,6 +20,7 @@
 #include "lflow.h"
 #include "lib/poll-loop.h"
 #include "ofctrl.h"
+#include "openvswitch/hmap.h"
 #include "openvswitch/match.h"
 #include "openvswitch/ofp-actions.h"
 #include "openvswitch/ofpbuf.h"
@@ -58,6 +59,15 @@ static struct simap localvif_to_ofport =
 SIMAP_INITIALIZER(_to_ofport);
 static struct hmap tunnels = HMAP_INITIALIZER();
 
+struct uuid_hash_node {
+struct hmap_node node;
+struct uuid uuid;
+};
+
+static struct hmap port_binding_uuids = HMAP_INITIALIZER(_binding_uuids);
+static struct hmap multicast_group_uuids =
+HMAP_INITIALIZER(_group_uuids);
+
 /* UUID to identify OF flows not associated with ovsdb rows. */
 static struct uuid *hc_uuid = NULL;
 static bool full_binding_processing = false;
@@ -636,6 +646,32 @@ consider_mc_group(enum mf_field_id mff_ovn_geneve,
 sset_destroy(_chassis);
 }
 
+static bool
+find_uuid_in_hmap(struct hmap *hmap_p, struct uuid *uuid) {
+struct uuid_hash_node *candidate;
+HMAP_FOR_EACH_WITH_HASH (candidate, node, uuid_hash(uuid), hmap_p) {
+if (uuid_equals(uuid, >uuid)) {
+return true;
+}
+}
+return false;
+}
+
+/* Deletes the flows whose UUIDs are in 'old' but not 'new', and then replaces
+ * 'old' by 'new'. */
+static void
+rationalize_hmap_and_delete_flows(struct hmap *old, struct hmap *new)
+{
+struct uuid_hash_node *uuid_node, *old_node;
+HMAP_FOR_EACH_SAFE (uuid_node, old_node, node, old) {
+if (!find_uuid_in_hmap(new, _node->uuid)) {
+ofctrl_remove_flows(_node->uuid);
+}
+}
+hmap_swap(old, new);
+hmap_clear(new);
+}
+
 void
 physical_run(struct controller_ctx *ctx, enum mf_field_id mff_ovn_geneve,
  const struct ovsrec_bridge *br_int, const char *this_chassis_id,
@@ -791,6 +827,8 @@ physical_run(struct controller_ctx *ctx, enum mf_field_id 
mff_ovn_geneve,
  * 64 for logical-to-physical translation. */
 const struct sbrec_port_binding *binding;
 if (full_binding_processing) {
+struct hmap new_port_binding_uuids =
+HMAP_INITIALIZER(_port_binding_uuids);
 SBREC_PORT_BINDING_FOR_EACH (binding, ctx->ovnsb_idl) {
 /* Because it is possible in the above code to enter this
  * for loop without having cleared the flow table first, we
@@ -798,7 +836,14 @@ physical_run(struct controller_ctx *ctx, enum mf_field_id 
mff_ovn_geneve,
 ofctrl_remove_flows(>header_.uuid);
 consider_port_binding(mff_ovn_geneve, ct_zones, local_datapaths,
   patched_datapaths, binding, );
-}
+struct uuid_hash_node *hash_node = xzalloc(sizeof *hash_node);
+hash_node->uuid = binding->header_.uuid;
+hmap_insert(_port_binding_uuids, _node->node,
+uuid_hash(_node->uuid));
+}
+rationalize_hmap_and_delete_flows(_binding_uuids,
+  _port_binding_uuids);
+hmap_destroy(_port_binding_uuids);
 full_binding_processing = false;
 } else {
 SBREC_PORT_BINDING_FOR_EACH_TRACKED (binding, ctx->ovnsb_idl) {
@@ -818,6 +863,8 @@ physical_run(struct controller_ctx *ctx, enum mf_field_id 
mff_ovn_geneve,
 const struct sbrec_multicast_group *mc;
 struct ofpbuf remote_ofpacts;
 ofpbuf_init(_ofpacts, 0);
+struct hmap new_multicast_group_uuids =
+HMAP_INITIALIZER(_multicast_group_uuids);
 SBREC_MULTICAST_GROUP_FOR_EACH (mc, ctx->ovnsb_idl) {
 /* As multicast groups are always reprocessed each time,
  * the first step is to clean the old flows for the group
@@ -825,7 +872,14 @@ physical_run(struct controller_ctx *ctx, enum mf_field_id 
mff_ovn_geneve,
 ofctrl_remove_flows(>header_.uuid);
 consider_mc_group(mff_ovn_geneve, ct_zones,
   local_datapaths, mc, , _ofpacts);
+struct uuid_hash_node *hash_node = xzalloc(sizeof *hash_node);
+hash_node->uuid = mc->header_.uuid;
+hmap_insert(_multicast_group_uuids, _node->node,
+uuid_hash(_node->uuid));
 }
+rationalize_hmap_and_delete_flows(_group_uuids,
+

Re: [ovs-dev] [PATCH rebase 3/3] ovn-controller: Use UDP checksums when creating Geneve tunnels.

2016-08-11 Thread Ryan Moats 
"dev" <dev-boun...@openvswitch.org> wrote on 08/11/2016 07:20:34 PM:

> From: Jesse Gross <je...@kernel.org>
> To: dev@openvswitch.org
> Date: 08/11/2016 07:22 PM
> Subject: [ovs-dev] [PATCH rebase 3/3] ovn-controller: Use UDP
> checksums when creating Geneve tunnels.
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> Currently metadata transmitted by OVN over Geneve tunnels is
> unprotected by any checksum other than the one provided by the link
> layer - this includes both the VNI and data stored in options. Turning
> on UDP checksums which cover this data has obvious benefits in terms of
> integrity protection.
>
> In terms of performance, this actually significantly increases throughput
> in most common cases when running on Linux based hosts without NICs
> supporting Geneve offload (around 60% for bulk traffic). The reason is
> that generally all NICs are capable of offloading transmitted and
received
> UDP checksums (viewed as ordinary UDP packets and not as tunnels). The
> benefit comes on the receive side where the validated outer UDP checksum
> can be used to additionally validate an inner checksum (such as TCP),
which
> in turn allows aggregation of packets to be more efficiently handled by
> the rest of the stack.
>
> Not all devices see such a benefit. The most notable exception is
hardware
> VTEPs (currently using VXLAN but potentially Geneve in the future). These
> devices are designed to not buffer entire packets in their switching
engines
> and are therefore unable to efficiently compute or validate UDP
checksums.
> In addition certain versions of the Linux kernel are not able to fully
> take advantage of Geneve capable NIC offloads in the presence of
checksums.
> (This is actually a pretty narrow corner case though - earlier versions
of
> Linux don't support Geneve offloads at all and later versions support
both
> offloads and checksums well.)
>
> In order avoid possible problems with these cases, efficient checksum
> receive performance is exposed as an encap option in the southbound
> database as a hint to remote senders. This currently defaults to off
> for hardware VTEPs and on for all other cases.
>
> Signed-off-by: Jesse Gross <je...@kernel.org>
> ---

Patch 2/3 never made my mailbox, so I'm using this to comment on both
patches

While I understand patch 2/3 does and I like, I'd like it even better if
it had test cases that would help me understand the changes that weren't
picked up by the original code, because that's a potential future
regression.

Assuming that is handled in a follow-on patch, consider this an ack on
both parts 2 and 3...

Acked-by: Ryan Moats <rmo...@us.ibm.com>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH rebase 1/3] ovn-controller: Fix memory leak when updating tunnels.

2016-08-11 Thread Ryan Moats 
"dev" <dev-boun...@openvswitch.org> wrote on 08/11/2016 07:20:32 PM:

> From: Jesse Gross <je...@kernel.org>
> To: dev@openvswitch.org
> Date: 08/11/2016 07:21 PM
> Subject: [ovs-dev] [PATCH rebase 1/3] ovn-controller: Fix memory
> leak when updating tunnels.
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> When a tunnel possibly needs to be updated, we are currently allocating
> a new name for it. This is not necessary and in fact nothing uses the
> name, which then results in the memory being leaked.
>
> Fixes: 1d45d5a9 ("ovn-controller: Change encaps_run to work
incrementally.")
> Signed-off-by: Jesse Gross <je...@kernel.org>
> ---

Good catch...

Acked-by: Ryan Moats <rmo...@us.ibm.com>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH v2] ovn-controller: Reset flow processing after (re)connection to switch

2016-08-11 Thread Ryan Moats 
"dev" <dev-boun...@openvswitch.org> wrote on 08/11/2016 07:21:39 AM:

> From: Numan Siddique <nusid...@redhat.com>
> To: ovs dev <dev@openvswitch.org>
> Date: 08/11/2016 07:21 AM
> Subject: [ovs-dev] [PATCH v2] ovn-controller: Reset flow processing
> after (re)connection to switch
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> When ovn-controller reconnects to the ovs-vswitchd, it deletes all the
> OF flows in the switch. It doesn't install the flows again, leaving
> the datapath broken unless ovn-controller is restarted or ovn-northd
> updates the SB DB.
>
> The reason for this is
>   - lflow_reset_processing() is not called after the reconnection
>   - the hmap "installed_flows" is not cleared, because of which
> ofctrl_put skips adding the flows to the switch.
>
> This patch fixes the issue and also adds a test case to test
> this scenario.
>
> Signed-off-by: Numan Siddique <nusid...@redhat.com>
> ---

This all looks does what it promises, so let's get the immediate
out of the way:

Acked-by: Ryan Moats <rmo...@us.ibm.com>

Having said that, I'm still concerned about what happens in the
kernel during an ovs-vswitchd restart...

I *think* everything will be fine for active connections whose
flow rules are still in the kernel, but there will be a momentary
loss of connectivity for those connections that require an upcall
to access the flows known to vswitchd.

If the above is right, then I'm willing to accept that, but it
begs two questions:

(1) is this something that is too complex for a unit test case?

(2) if it isn't too complex, how would one code such a unit
test case?

___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH] ovs-bugtool: Switch from MD5 to SHA-256.

2016-08-11 Thread Ryan Moats 


"dev" <dev-boun...@openvswitch.org> wrote on 08/10/2016 11:14:27 PM:

> From: Ben Pfaff <b...@ovn.org>
> To: dev@openvswitch.org
> Cc: Ben Pfaff <b...@ovn.org>
> Date: 08/10/2016 11:14 PM
> Subject: [ovs-dev] [PATCH] ovs-bugtool: Switch from MD5 to SHA-256.
> Sent by: "dev" <dev-boun...@openvswitch.org>
>
> While going through a FIPS certification process we discovered that
> ovs-bugtool uses MD5 to identify the contents of files.  FIPS doesn't
allow
> use of the obsolete and broken MD5 algorithm, so this commit switches to
> SHA-256.
>
> In a way, this is a silly requirement.  ovs-bugtool only uses MD5 to
> identify file content, mostly to ensure that the contents of the bug
report
> have not been corrupted.  MD5 is perfectly adequate for that purpose; in
> fact a 16-bit CRC would probably be adequate.  On the other hand, there
is
> basically no cost and no disadvantage to switching to SHA-256, so why not
> do it?  That's why I think that this is a reasonable change.
>
> VMware-BZ: #1708786
> Signed-off-by: Ben Pfaff <b...@ovn.org>
> ---

Yes, it's annoying, but arguing with FIPS reminds me of bringing a knife
to a gun fight...

The patch looks sane so ...

Acked-by: Ryan Moats <rmo...@us.ibm.com>

(On a side note, I realized that we don't really have a unit test for this
tool, but honestly, I'm not sure it's worth adding one - I leave that to
wiser heads than mine...)
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

  1   2   3   4   5   6   7   8   >