Re: [ovs-dev] [PATCH v2 2/2] conntrack: Key connections by zone.
To seperate into N cmaps, why not use hash value divided by N? Simon Horman 于2024年5月1日周三 19:06写道： > On Wed, Apr 24, 2024 at 02:44:54PM +0200, Felix Huettner via dev wrote: > > Currently conntrack uses a single large cmap for all connections stored. > > This cmap contains all connections for all conntrack zones which are > > completely separate from each other. By separating each zone to its own > > cmap we can significantly optimize the performance when using multiple > > zones. > > > > The change fixes a similar issue as [1] where slow conntrack zone flush > > operations significantly slow down OVN router failover. The difference is > > just that this fix is used whith dpdk, while [1] was when using the ovs > > kernel module. > > > > As we now need to store more cmap's the memory usage of struct conntrack > > increases by 524280 bytes. Additionally we need 65535 cmaps with 128 > > bytes each. This leads to a total memory increase of around 10MB. > > > > Running "./ovstest test-conntrack benchmark 4 33554432 32 1" shows no > > real difference in the multithreading behaviour against a single zone. > > > > Running the new "./ovstest test-conntrack benchmark-zones" show > > significant speedups as shown below. The values for "ct execute" are for > > acting on the complete zone with all its entries in total (so in the > > first case adding 10,000 new conntrack entries). All tests are run 1000 > > times. > > > > When running with 1,000 zones with 10,000 entries each we see the > > following results (all in microseconds): > > "./ovstest test-conntrack benchmark-zones 1 1000 1000" > > > > +--++-+-+ > > | Min | Max | 95%ile | Avg | > > ++--++-+-+ > > | ct execute (commit)| || | | > > |with commit | 2266 | 3505 | 2707.06 | 2592.06 | > > | without commit | 2411 | 12730 | 4432.50 | 2736.78 | > > ++--++-+-+ > > | ct execute (no commit) | || | | > > |with commit | 699 | 1238 | 886.15 | 722.67 | > > | without commit | 700 | 3377 | 1934.42 | 803.53 | > > ++--++-+-+ > > | flush full zone| || | | > > |with commit | 619 | 1122 | 901.36 | 679.15 | > > | without commit | 618 | 105078 | 64591 | 2886.46 | > > ++--++-+-+ > > | flush empty zone | || | | > > |with commit |0 | 5 |1.00 |0.64 | > > | without commit | 54 | 87469 | 64520 | 2172.25 | > > ++--++-+-+ > > > > When running with 10,000 zones with 1,000 entries each we see the > > following results (all in microseconds): > > "./ovstest test-conntrack benchmark-zones 1000 1 1000" > > > > +--++-+-+ > > | Min | Max | 95%ile | Avg | > > ++--++-+-+ > > | ct execute (commit)| || | | > > |with commit | 215 |287 | 231.88 | 222.30 | > > | without commit | 214 | 1692 | 569.18 | 285.83 | > > ++--++-+-+ > > | ct execute (no commit) | || | | > > |with commit | 68 | 97 | 74.69 | 70.09 | > > | without commit | 68 |300 | 158.40 | 82.06 | > > ++--++-+-+ > > | flush full zone| || | | > > |with commit | 47 |211 | 56.34 | 50.34 | > > | without commit | 48 | 96330 | 63392 | 63923 | > > ++--++-+-+ > > | flush empty zone | || | | > > |with commit |0 | 1 |1.00 |0.44 | > > | without commit |3 | 109728 | 63923 | 3629.44 | > > ++--++-+-+ > > > > Comparing the averages we see: > > * a moderate performance improvement for conntrack_execute with or > > without commiting of around 6% to 23% > > * a significant performance improvement for flushing a full zone of > > around 75% to 99% > > * an even more significant improvement for flushing empty zones since we > > no longer need to check any unrelated connections > > Very nice numbers indeed. > > > [1] 9ec849e8aa869b646c372fac552ae2609a4b5f66 > > > > Signed-off-by: Felix Huettner > > Acked-by: Simon Horman > > ... > > ___ > dev mailing list > d...@openvswitch.org >
Re: [ovs-dev] [PATCH] dpif-netdev: Fix length calculation of netdet_flow_key.
Hi, Eelco,
No hurry, take your time!
Eelco Chaudron 于2023年8月16日周三 16:43写道：
>
>
> On 16 Aug 2023, at 4:00, Peng He wrote:
>
> > Hi, Eelco,
> >
> > could you take a look at this patch?
>
> Hi Peng, yes it’s on my TODO list, but it might be a while as I’m
> recovering from three weeks of PTO ;)
>
> //Eelco
>
>
> >
> > Zhiqi Chen via dev 于2023年8月13日周日 17:09写道：
> >
> >> The 'len' of a netdev_flow_key initialized by netdev_flow_key_init()
> >> is always zero, which may cause errors when cloning a netdev_flow_key
> >> by netdev_flow_key_clone().
> >>
> >> Currently the 'len' member of a netdev_flow_key initialized by
> >> netdev_flow_key_init() is not used, so this error will not cause any
> >> bad behavior for now.
> >>
> >> Fixes: c82f496c3b69 ("dpif-netdev: Use unmasked key when adding datapath
> >> flows.")
> >> Signed-off-by: Zhiqi Chen
> >> ---
> >> lib/dpif-netdev.c | 3 +--
> >> 1 file changed, 1 insertion(+), 2 deletions(-)
> >>
> >> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> >> index 0b623fcea..0d8b1f4ae 100644
> >> --- a/lib/dpif-netdev.c
> >> +++ b/lib/dpif-netdev.c
> >> @@ -3380,14 +3380,13 @@ static inline void
> >> netdev_flow_key_init(struct netdev_flow_key *key,
> >> const struct flow *flow)
> >> {
> >> -uint64_t *dst = miniflow_values(>mf);
> >> uint32_t hash = 0;
> >> uint64_t value;
> >>
> >> miniflow_map_init(>mf, flow);
> >> miniflow_init(>mf, flow);
> >>
> >> -size_t n = dst - miniflow_get_values(>mf);
> >> +size_t n = miniflow_n_values(>mf);
> >>
> >> FLOW_FOR_EACH_IN_MAPS (value, flow, key->mf.map) {
> >> hash = hash_add64(hash, value);
> >> --
> >> 2.39.2 (Apple Git-143)
> >>
> >> ___
> >> dev mailing list
> >> d...@openvswitch.org
> >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >>
> >
> >
> > --
> > hepeng
>
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH] dpif-netdev: Fix length calculation of netdet_flow_key.
Hi, Eelco,
could you take a look at this patch?
Zhiqi Chen via dev 于2023年8月13日周日 17:09写道：
> The 'len' of a netdev_flow_key initialized by netdev_flow_key_init()
> is always zero, which may cause errors when cloning a netdev_flow_key
> by netdev_flow_key_clone().
>
> Currently the 'len' member of a netdev_flow_key initialized by
> netdev_flow_key_init() is not used, so this error will not cause any
> bad behavior for now.
>
> Fixes: c82f496c3b69 ("dpif-netdev: Use unmasked key when adding datapath
> flows.")
> Signed-off-by: Zhiqi Chen
> ---
> lib/dpif-netdev.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> index 0b623fcea..0d8b1f4ae 100644
> --- a/lib/dpif-netdev.c
> +++ b/lib/dpif-netdev.c
> @@ -3380,14 +3380,13 @@ static inline void
> netdev_flow_key_init(struct netdev_flow_key *key,
> const struct flow *flow)
> {
> -uint64_t *dst = miniflow_values(>mf);
> uint32_t hash = 0;
> uint64_t value;
>
> miniflow_map_init(>mf, flow);
> miniflow_init(>mf, flow);
>
> -size_t n = dst - miniflow_get_values(>mf);
> +size_t n = miniflow_n_values(>mf);
>
> FLOW_FOR_EACH_IN_MAPS (value, flow, key->mf.map) {
> hash = hash_add64(hash, value);
> --
> 2.39.2 (Apple Git-143)
>
> ___
> dev mailing list
> d...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v12] ofproto-dpif-upcall: fix push_dp_ops
Thanks Eelco Chaudron 于2023年8月16日 周三00:02写道： > > > On 15 Aug 2023, at 13:18, Eelco Chaudron wrote: > > > On 15 Aug 2023, at 11:06, Peng He wrote: > > > >> Do I need to send a new version to fix the indentation? > > > > Nope, I’ll fix it when committing the patch. Will be later this week, as > I need to catch up on some email etc. > > Thanks for the patch Peng, I’ve fixed the indentation and committed the > patch. > > Cheers, > > Eelco > > > >> Eelco Chaudron 于2023年8月15日周二 15:40写道： > >> > >>> > >>> > >>> On 27 Jul 2023, at 12:53, Ilya Maximets wrote: > >>> > >>>> On 7/27/23 04:16, Peng He wrote: > >>>>> > >>>>> > >>>>> Simon Horman >>> simon.hor...@corigine.com>> 于2023年7月27日周四 01:04写道： > >>>>> > >>>>> On Thu, Jul 06, 2023 at 04:59:36PM +0800, Peng He wrote: > >>>>> > >>>>> ... > >>>>> > >>>>> > >>>> +dnl Replace OpenFlow rules, trigger revalidation and wait > >>> for it to > >>>>> > >>> complete. > >>>>> > >>>> +AT_CHECK([echo 'table=0,in_port=p1,ip > actions=ct(commit)' | > >>> ovs-ofctl > >>>>> > >>> --bundle replace-flows br0 -]) > >>>>> > >>>> +AT_CHECK([ovs-appctl revalidator/wait]) > >>>>> > >>>> + > >>>>> > >>>> +dnl Inconsistent ukey should be deleted. > >>>>> > >>>> +AT_CHECK([ovs-appctl upcall/show | grep keys | grep -q -v > >>> 0], [1]) > >>>>> > >>>> + > >>>>> > >>>> +dnl Check the log for the flow modification error. > >>>>> > >>>> +AT_CHECK([grep -q -E ".*failed to put.*$" > ovs-vswitchd.log]) > >>>>> > >>>> + > >>>>> > >>>> +dnl Remove warning logs to let test suite pass. > >>>>> > >>>> +OVS_VSWITCHD_STOP(["dnl > >>>>> > >>>> +/.*failed to put.*$/d > >>>>> > >>>> +/.*failed to flow_del.*$/d"]) > >>>>> > >>> > >>>>> > >>> You missed the indentation suggested by Ilya: > >>>>> > >>> > >>>>> > >>> > >>>>> > >> Which kind of the email client are you using... > >>>>> > >> The indentation here is really easy to miss > >>>>> > > > >>>>> > > From the User-Agent data, he seems to be using > >>> Thunderbird/102.10.0 but the emails are plain text, so the problem is > >>> probably with your email client converting them to “html” like format. > If > >>> you can configure a fixed width font for text emails you might spot > these > >>> thinks easier. > >>>>> > > >>>>> > > >>>>> > I change to use another email client now, thanks ! > >>>>> > >>>>> Hi Peng He, all, > >>>>> > >>>>> Am I correct in assuming that there will be a v13? > >>>>> > >>>>> > >>>>> Hi, if need, I can submit a v13, the current version is just missing > >>> some indentation. > >>>>> > >>>>> "The changes look good, with one little nit (see below), but those > can > >>> be applied during commit." > >>>>> I thought it would be changed by the maintainer during merging... > >>>> > >>>> My impression was that Eelco was generally happy with the patch, > >>>> and the small indentation fix for a test can be applied on commit. > >>>> > >>>> But Eelco is on PTO for now, and I didn't get to review that patch > >>>> myself yet in order to accept it. > >>> > >>> Hi Simon/Peng, > >>> > >>> Sorry for the confusion :) I’m back from PTO, and I can apply the > patch, > >>> unless Simon has some objections? > >>> > >>> Cheers, > >>> > >>> Eelco > >>> > >>> > >> > >> -- > >> hepeng > > -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v12] ofproto-dpif-upcall: fix push_dp_ops
Ok, thanks. Eelco Chaudron 于2023年8月15日周二 19:19写道： > > > On 15 Aug 2023, at 11:06, Peng He wrote: > > > Do I need to send a new version to fix the indentation? > > Nope, I’ll fix it when committing the patch. Will be later this week, as I > need to catch up on some email etc. > > //Eelco > > > Eelco Chaudron 于2023年8月15日周二 15:40写道： > > > >> > >> > >> On 27 Jul 2023, at 12:53, Ilya Maximets wrote: > >> > >>> On 7/27/23 04:16, Peng He wrote: > >>>> > >>>> > >>>> Simon Horman >> simon.hor...@corigine.com>> 于2023年7月27日周四 01:04写道： > >>>> > >>>> On Thu, Jul 06, 2023 at 04:59:36PM +0800, Peng He wrote: > >>>> > >>>> ... > >>>> > >>>> > >>>> +dnl Replace OpenFlow rules, trigger revalidation and wait > >> for it to > >>>> > >>> complete. > >>>> > >>>> +AT_CHECK([echo 'table=0,in_port=p1,ip actions=ct(commit)' > | > >> ovs-ofctl > >>>> > >>> --bundle replace-flows br0 -]) > >>>> > >>>> +AT_CHECK([ovs-appctl revalidator/wait]) > >>>> > >>>> + > >>>> > >>>> +dnl Inconsistent ukey should be deleted. > >>>> > >>>> +AT_CHECK([ovs-appctl upcall/show | grep keys | grep -q -v > >> 0], [1]) > >>>> > >>>> + > >>>> > >>>> +dnl Check the log for the flow modification error. > >>>> > >>>> +AT_CHECK([grep -q -E ".*failed to put.*$" > ovs-vswitchd.log]) > >>>> > >>>> + > >>>> > >>>> +dnl Remove warning logs to let test suite pass. > >>>> > >>>> +OVS_VSWITCHD_STOP(["dnl > >>>> > >>>> +/.*failed to put.*$/d > >>>> > >>>> +/.*failed to flow_del.*$/d"]) > >>>> > >>> > >>>> > >>> You missed the indentation suggested by Ilya: > >>>> > >>> > >>>> > >>> > >>>> > >> Which kind of the email client are you using... > >>>> > >> The indentation here is really easy to miss > >>>> > > > >>>> > > From the User-Agent data, he seems to be using > >> Thunderbird/102.10.0 but the emails are plain text, so the problem is > >> probably with your email client converting them to “html” like format. > If > >> you can configure a fixed width font for text emails you might spot > these > >> thinks easier. > >>>> > > >>>> > > >>>> > I change to use another email client now, thanks ! > >>>> > >>>> Hi Peng He, all, > >>>> > >>>> Am I correct in assuming that there will be a v13? > >>>> > >>>> > >>>> Hi, if need, I can submit a v13, the current version is just missing > >> some indentation. > >>>> > >>>> "The changes look good, with one little nit (see below), but those can > >> be applied during commit." > >>>> I thought it would be changed by the maintainer during merging... > >>> > >>> My impression was that Eelco was generally happy with the patch, > >>> and the small indentation fix for a test can be applied on commit. > >>> > >>> But Eelco is on PTO for now, and I didn't get to review that patch > >>> myself yet in order to accept it. > >> > >> Hi Simon/Peng, > >> > >> Sorry for the confusion :) I’m back from PTO, and I can apply the patch, > >> unless Simon has some objections? > >> > >> Cheers, > >> > >> Eelco > >> > >> > > > > -- > > hepeng > > -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v12] ofproto-dpif-upcall: fix push_dp_ops
Do I need to send a new version to fix the indentation? Eelco Chaudron 于2023年8月15日周二 15:40写道： > > > On 27 Jul 2023, at 12:53, Ilya Maximets wrote: > > > On 7/27/23 04:16, Peng He wrote: > >> > >> > >> Simon Horman simon.hor...@corigine.com>> 于2023年7月27日周四 01:04写道： > >> > >> On Thu, Jul 06, 2023 at 04:59:36PM +0800, Peng He wrote: > >> > >> ... > >> > >> > >>>> +dnl Replace OpenFlow rules, trigger revalidation and wait > for it to > >> > >>> complete. > >> > >>>> +AT_CHECK([echo 'table=0,in_port=p1,ip actions=ct(commit)' | > ovs-ofctl > >> > >>> --bundle replace-flows br0 -]) > >> > >>>> +AT_CHECK([ovs-appctl revalidator/wait]) > >> > >>>> + > >> > >>>> +dnl Inconsistent ukey should be deleted. > >> > >>>> +AT_CHECK([ovs-appctl upcall/show | grep keys | grep -q -v > 0], [1]) > >> > >>>> + > >> > >>>> +dnl Check the log for the flow modification error. > >> > >>>> +AT_CHECK([grep -q -E ".*failed to put.*$" ovs-vswitchd.log]) > >> > >>>> + > >> > >>>> +dnl Remove warning logs to let test suite pass. > >> > >>>> +OVS_VSWITCHD_STOP(["dnl > >> > >>>> +/.*failed to put.*$/d > >> > >>>> +/.*failed to flow_del.*$/d"]) > >> > >>> > >> > >>> You missed the indentation suggested by Ilya: > >> > >>> > >> > >>> > >> > >> Which kind of the email client are you using... > >> > >> The indentation here is really easy to miss > >> > > > >> > > From the User-Agent data, he seems to be using > Thunderbird/102.10.0 but the emails are plain text, so the problem is > probably with your email client converting them to “html” like format. If > you can configure a fixed width font for text emails you might spot these > thinks easier. > >> > > >> > > >> > I change to use another email client now, thanks ! > >> > >> Hi Peng He, all, > >> > >> Am I correct in assuming that there will be a v13? > >> > >> > >> Hi, if need, I can submit a v13, the current version is just missing > some indentation. > >> > >> "The changes look good, with one little nit (see below), but those can > be applied during commit." > >> I thought it would be changed by the maintainer during merging... > > > > My impression was that Eelco was generally happy with the patch, > > and the small indentation fix for a test can be applied on commit. > > > > But Eelco is on PTO for now, and I didn't get to review that patch > > myself yet in order to accept it. > > Hi Simon/Peng, > > Sorry for the confusion :) I’m back from PTO, and I can apply the patch, > unless Simon has some objections? > > Cheers, > > Eelco > > -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v5] dpif-netdev: fix dpif_netdev_flow_put
Thanks！
Ilya Maximets 于2023年8月15日 周二02:02写道：
> On 8/14/23 04:37, Peng He wrote:
> > OVS allows overlapping megaflows, as long as the actions of these
> > megaflows are equal. However, the current implementation of action
> > modification relies on flow_lookup instead of ufid, this could result
> > in looking up a wrong megaflow and make the ukeys and megaflows
> inconsistent
> >
> > Just like the test case in the patch, at first we have a rule with the
> > prefix:
> >
> > 10.1.2.0/24
> >
> > and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with
> IP
> > 10.1.2.2 is received.
> >
> > Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep
> the
> > 10.1.2.2/24 megaflow and just changes its action instead of extending
> > the prefix into 10.1.2.2/16.
> >
> > then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
> > this time, we will have an overlapping megaflow with the right prefix:
> > 10.1.0.2/16
> >
> > now we have two megaflows:
> > 10.1.2.2/24
> > 10.1.0.2/16
> >
> > last, suppose we have changed the ruleset again. The revalidator this
> > time still decides to change the actions of both megaflows instead of
> > deleting them.
> >
> > The dpif_netdev_flow_put will search the megaflow to modify with unmasked
> > keys, however it might lookup the wrong megaflow as the key 10.1.2.2
> matches
> > both 10.1.2.2/24 and 10.1.0.2/16!
> >
> > This patch changes the megaflow lookup code in modification path into
> > relying the ufid to find the correct megaflow instead of key lookup.
> >
> > Fixes: beb75a40fdc2 ("userspace: Switching of L3 packets in L2 pipeline")
> > Signed-off-by: Peng He
> > ---
> > lib/dpif-netdev.c | 45 ++---
> > tests/pmd.at | 47 +++
> > 2 files changed, 77 insertions(+), 15 deletions(-)
>
> Thanks! Applied and backported down to 2.17.
>
> Best regards, Ilya Maximets.
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v5] dpif-netdev: fix dpif_netdev_flow_put
OVS allows overlapping megaflows, as long as the actions of these
megaflows are equal. However, the current implementation of action
modification relies on flow_lookup instead of ufid, this could result
in looking up a wrong megaflow and make the ukeys and megaflows inconsistent
Just like the test case in the patch, at first we have a rule with the
prefix:
10.1.2.0/24
and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with IP
10.1.2.2 is received.
Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep the
10.1.2.2/24 megaflow and just changes its action instead of extending
the prefix into 10.1.2.2/16.
then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
this time, we will have an overlapping megaflow with the right prefix:
10.1.0.2/16
now we have two megaflows:
10.1.2.2/24
10.1.0.2/16
last, suppose we have changed the ruleset again. The revalidator this
time still decides to change the actions of both megaflows instead of
deleting them.
The dpif_netdev_flow_put will search the megaflow to modify with unmasked
keys, however it might lookup the wrong megaflow as the key 10.1.2.2 matches
both 10.1.2.2/24 and 10.1.0.2/16!
This patch changes the megaflow lookup code in modification path into
relying the ufid to find the correct megaflow instead of key lookup.
Fixes: beb75a40fdc2 ("userspace: Switching of L3 packets in L2 pipeline")
Signed-off-by: Peng He
---
lib/dpif-netdev.c | 45 ++---
tests/pmd.at | 47 +++
2 files changed, 77 insertions(+), 15 deletions(-)
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index 70b953ae6..8479630b8 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -4191,7 +4191,7 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
const struct dpif_flow_put *put,
struct dpif_flow_stats *stats)
{
-struct dp_netdev_flow *netdev_flow;
+struct dp_netdev_flow *netdev_flow = NULL;
int error = 0;
if (stats) {
@@ -4199,16 +4199,35 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
}
ovs_mutex_lock(>flow_mutex);
-netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
-if (!netdev_flow) {
-if (put->flags & DPIF_FP_CREATE) {
-dp_netdev_flow_add(pmd, match, ufid, put->actions,
- put->actions_len, ODPP_NONE);
+if (put->ufid) {
+netdev_flow = dp_netdev_pmd_find_flow(pmd, put->ufid,
+ put->key, put->key_len);
+} else {
+/* Use key instead of the locally generated ufid
+ * to search netdev_flow. */
+netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
+}
+
+if (put->flags & DPIF_FP_CREATE) {
+if (!netdev_flow) {
+dp_netdev_flow_add(pmd, match, ufid,
+ put->actions, put->actions_len, ODPP_NONE);
} else {
-error = ENOENT;
+error = EEXIST;
}
-} else {
-if (put->flags & DPIF_FP_MODIFY) {
+goto exit;
+}
+
+if (put->flags & DPIF_FP_MODIFY) {
+if (!netdev_flow) {
+error = ENOENT;
+} else {
+if (!put->ufid && !flow_equal(>flow, _flow->flow)) {
+/* Overlapping flow. */
+error = EINVAL;
+goto exit;
+}
+
struct dp_netdev_actions *new_actions;
struct dp_netdev_actions *old_actions;
@@ -4239,15 +4258,11 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
* counter, and subtracting it before outputting the stats */
error = EOPNOTSUPP;
}
-
ovsrcu_postpone(dp_netdev_actions_free, old_actions);
-} else if (put->flags & DPIF_FP_CREATE) {
-error = EEXIST;
-} else {
-/* Overlapping flow. */
-error = EINVAL;
}
}
+
+exit:
ovs_mutex_unlock(>flow_mutex);
return error;
}
diff --git a/tests/pmd.at b/tests/pmd.at
index 48f3d432d..3d354dc41 100644
--- a/tests/pmd.at
+++ b/tests/pmd.at
@@ -1300,3 +1300,50 @@ OVS_WAIT_UNTIL([tail -n +$LINENUM ovs-vswitchd.log |
grep "PMD load based sleeps
OVS_VSWITCHD_STOP
AT_CLEANUP
+
+AT_SETUP([PMD - revalidator wrongly modify userspace megaflows])
+
+OVS_VSWITCHD_START(
+[add-port br0 p1 \
+ -- set bridge br0 datapath-type=dummy \
+ -- set interface p1 type=dummy-pmd \
+ -- add-port br0 p2 \
+ -- set interface p2 type=dummy-pmd
+], [], [], [DUMMY_NUMA])
+
+dnl Add one openflow rule and generate a megaflow.
+AT_CHECK([ovs-ofctl add-flow br0
'table=0,in_port=p1,ip,nw_dst=10.1.2.0/24,actions=p2'])
+AT_CHECK([ovs-appctl netdev-dummy/receive p1
'ipv4(src=10.0.0.1,dst=10.1.2.2,proto=6),tcp(src=1,dst=2)'])
+
+OVS_WAIT_UNTIL_EQUAL([o
Re: [ovs-dev] [ovs-dev v4] dpif-netdev: fix dpif_netdev_flow_put
Ilya Maximets 于2023年8月12日周六 02:18写道：
> On 8/11/23 19:37, Ilya Maximets wrote:
> > On 8/9/23 05:02, Peng He wrote:
> >> OVS allows overlapping megaflows, as long as the actions of these
> >> megaflows are equal. However, the current implementation of action
> >> modification relies on flow_lookup instead of ufid, this could result
> >> in looking up a wrong megaflow and make the ukeys and megaflows
> inconsistent
> >>
> >> Just like the test case in the patch, at first we have a rule with the
> >> prefix:
> >>
> >> 10.1.2.0/24
> >>
> >> and we will get a megaflow with prefixes 10.1.2.2/24 when a packet
> with IP
> >> 10.1.2.2 is received.
> >>
> >> Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep
> the
> >> 10.1.2.2/24 megaflow and just changes its action instead of extending
> >> the prefix into 10.1.2.2/16.
> >>
> >> then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
> >> this time, we will have an overlapping megaflow with the right prefix:
> >> 10.1.0.2/16
> >>
> >> now we have two megaflows:
> >> 10.1.2.2/24
> >> 10.1.0.2/16
> >>
> >> last, suppose we have changed the ruleset again. The revalidator this
> >> time still decides to change the actions of both megaflows instead of
> >> deleting them.
> >>
> >> The dpif_netdev_flow_put will search the megaflow to modify with
> unmasked
> >> keys, however it might lookup the wrong megaflow as the key 10.1.2.2
> matches
> >> both 10.1.2.2/24 and 10.1.0.2/16!
> >>
> >> This patch changes the megaflow lookup code in modification path into
> >> relying the ufid to find the correct megaflow instead of key lookup.
> >>
> >> Signed-off-by: Peng He
> >> ---
> >> lib/dpif-netdev.c | 45 ++---
> >> tests/pmd.at | 47 +++
> >> 2 files changed, 77 insertions(+), 15 deletions(-)
> >>
> >> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> >> index 70b953ae6..8479630b8 100644
> >> --- a/lib/dpif-netdev.c
> >> +++ b/lib/dpif-netdev.c
> >> @@ -4191,7 +4191,7 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
> >> const struct dpif_flow_put *put,
> >> struct dpif_flow_stats *stats)
> >> {
> >> -struct dp_netdev_flow *netdev_flow;
> >> +struct dp_netdev_flow *netdev_flow = NULL;
> >> int error = 0;
> >>
> >> if (stats) {
> >> @@ -4199,16 +4199,35 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread
> *pmd,
> >> }
> >>
> >> ovs_mutex_lock(>flow_mutex);
> >> -netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> >> -if (!netdev_flow) {
> >> -if (put->flags & DPIF_FP_CREATE) {
> >> -dp_netdev_flow_add(pmd, match, ufid, put->actions,
> >> - put->actions_len, ODPP_NONE);
> >> +if (put->ufid) {
> >> +netdev_flow = dp_netdev_pmd_find_flow(pmd, put->ufid,
> >> + put->key, put->key_len);
> >> +} else {
> >> +/* Use key instead of the locally generated ufid
> >> + * to search netdev_flow. */
> >> +netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> >> +}
> >> +
> >> +if (put->flags & DPIF_FP_CREATE) {
> >> +if (!netdev_flow) {
> >> +dp_netdev_flow_add(pmd, match, ufid,
> >> + put->actions, put->actions_len,
> ODPP_NONE);
> >> } else {
> >> -error = ENOENT;
> >> +error = EEXIST;
> >> }
> >> -} else {
> >> -if (put->flags & DPIF_FP_MODIFY) {
> >> +goto exit;
> >> +}
> >> +
> >> +if (put->flags & DPIF_FP_MODIFY) {
> >> +if (!netdev_flow) {
> >> +error = ENOENT;
> >> +} else {
> >> +if (!put->ufid && !flow_equal(>flow,
> _flow->flow)) {
> >> +/* Overlapping flow. */
> >> +error = EINVAL;
> >> +goto exit;
> >> +}
> >> +
> >> struct dp_netd
Re: [ovs-dev] [ovs-dev v4] dpif-netdev: fix dpif_netdev_flow_put
again, I forget the fixes tag ... 0-day Robot 于2023年8月9日周三 11:19写道： > Bleep bloop. Greetings Peng He, I am a robot and I have tried out your > patch. > Thanks for your contribution. > > I encountered some error that I wasn't expecting. See the details below. > > > checkpatch: > ERROR: Author Peng He needs to sign off. > WARNING: Unexpected sign-offs from developers who are not authors or > co-authors or committers: Peng He > Lines checked: 181, Warnings: 1, Errors: 1 > > > Please check this out. If you feel there has been an error, please email > acon...@redhat.com > > Thanks, > 0-day Robot > -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v4] dpif-netdev: fix dpif_netdev_flow_put
OVS allows overlapping megaflows, as long as the actions of these
megaflows are equal. However, the current implementation of action
modification relies on flow_lookup instead of ufid, this could result
in looking up a wrong megaflow and make the ukeys and megaflows inconsistent
Just like the test case in the patch, at first we have a rule with the
prefix:
10.1.2.0/24
and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with IP
10.1.2.2 is received.
Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep the
10.1.2.2/24 megaflow and just changes its action instead of extending
the prefix into 10.1.2.2/16.
then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
this time, we will have an overlapping megaflow with the right prefix:
10.1.0.2/16
now we have two megaflows:
10.1.2.2/24
10.1.0.2/16
last, suppose we have changed the ruleset again. The revalidator this
time still decides to change the actions of both megaflows instead of
deleting them.
The dpif_netdev_flow_put will search the megaflow to modify with unmasked
keys, however it might lookup the wrong megaflow as the key 10.1.2.2 matches
both 10.1.2.2/24 and 10.1.0.2/16!
This patch changes the megaflow lookup code in modification path into
relying the ufid to find the correct megaflow instead of key lookup.
Signed-off-by: Peng He
---
lib/dpif-netdev.c | 45 ++---
tests/pmd.at | 47 +++
2 files changed, 77 insertions(+), 15 deletions(-)
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index 70b953ae6..8479630b8 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -4191,7 +4191,7 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
const struct dpif_flow_put *put,
struct dpif_flow_stats *stats)
{
-struct dp_netdev_flow *netdev_flow;
+struct dp_netdev_flow *netdev_flow = NULL;
int error = 0;
if (stats) {
@@ -4199,16 +4199,35 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
}
ovs_mutex_lock(>flow_mutex);
-netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
-if (!netdev_flow) {
-if (put->flags & DPIF_FP_CREATE) {
-dp_netdev_flow_add(pmd, match, ufid, put->actions,
- put->actions_len, ODPP_NONE);
+if (put->ufid) {
+netdev_flow = dp_netdev_pmd_find_flow(pmd, put->ufid,
+ put->key, put->key_len);
+} else {
+/* Use key instead of the locally generated ufid
+ * to search netdev_flow. */
+netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
+}
+
+if (put->flags & DPIF_FP_CREATE) {
+if (!netdev_flow) {
+dp_netdev_flow_add(pmd, match, ufid,
+ put->actions, put->actions_len, ODPP_NONE);
} else {
-error = ENOENT;
+error = EEXIST;
}
-} else {
-if (put->flags & DPIF_FP_MODIFY) {
+goto exit;
+}
+
+if (put->flags & DPIF_FP_MODIFY) {
+if (!netdev_flow) {
+error = ENOENT;
+} else {
+if (!put->ufid && !flow_equal(>flow, _flow->flow)) {
+/* Overlapping flow. */
+error = EINVAL;
+goto exit;
+}
+
struct dp_netdev_actions *new_actions;
struct dp_netdev_actions *old_actions;
@@ -4239,15 +4258,11 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
* counter, and subtracting it before outputting the stats */
error = EOPNOTSUPP;
}
-
ovsrcu_postpone(dp_netdev_actions_free, old_actions);
-} else if (put->flags & DPIF_FP_CREATE) {
-error = EEXIST;
-} else {
-/* Overlapping flow. */
-error = EINVAL;
}
}
+
+exit:
ovs_mutex_unlock(>flow_mutex);
return error;
}
diff --git a/tests/pmd.at b/tests/pmd.at
index 48f3d432d..b03cd831e 100644
--- a/tests/pmd.at
+++ b/tests/pmd.at
@@ -1300,3 +1300,50 @@ OVS_WAIT_UNTIL([tail -n +$LINENUM ovs-vswitchd.log |
grep "PMD load based sleeps
OVS_VSWITCHD_STOP
AT_CLEANUP
+
+AT_SETUP([PMD - revalidator wrongly modify userspace megaflows])
+
+OVS_VSWITCHD_START(
+[add-port br0 p1 \
+ -- set bridge br0 datapath-type=dummy \
+ -- set interface p1 type=dummy-pmd \
+ -- add-port br0 p2 \
+ -- set interface p2 type=dummy-pmd
+])
+
+dnl Add one openflow rule and generate a megaflow.
+AT_CHECK([ovs-ofctl add-flow br0
'table=0,in_port=p1,ip,nw_dst=10.1.2.0/24,actions=p2'])
+AT_CHECK([ovs-appctl netdev-dummy/receive p1
'ipv4(src=10.0.0.1,dst=10.1.2.2),tcp(src=1,dst=2)'])
+
+OVS_WAIT_UNTIL_EQUAL([ovs-appctl dpctl/dump-flows | sed 's/.*core:
[[0-9]]*//'], [
+recirc_id(0),in_port(1),packet_type(ns=0,id=0),eth
Re: [ovs-dev] [ovs-dev v3] dpif-netdev: fix dpif_netdev_flow_put
Ilya Maximets 于2023年8月8日周二 07:30写道：
> On 7/31/23 06:55, Peng He wrote:
> > OVS allows overlapping megaflows, as long as the actions of these
> > megaflows are equal. However, the current implementation of action
> > modification relies on flow_lookup instead of ufid, this could result
> > in looking up a wrong megaflow and make the ukeys and megaflows
> inconsistent
> >
> > Just like the test case in the patch, at first we have a rule with the
> > prefix:
> >
> > 10.1.2.0/24
> >
> > and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with
> IP
> > 10.1.2.2 is received.
> >
> > Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep
> the
> > 10.1.2.2/24 megaflow and just changes its action instead of extending
> > the prefix into 10.1.2.2/16.
> >
> > then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
> > this time, we will have an overlapping megaflow with the right prefix:
> > 10.1.0.2/16
> >
> > now we have two megaflows:
> > 10.1.2.2/24
> > 10.1.0.2/16
> >
> > last, suppose we have changed the ruleset again. The revalidator this
> > time still decides to change the actions of both megaflows instead of
> > deleting them.
> >
> > The dpif_netdev_flow_put will search the megaflow to modify with unmasked
> > keys, however it might lookup the wrong megaflow as the key 10.1.2.2
> matches
> > both 10.1.2.2/24 and 10.1.0.2/16!
> >
> > This patch changes the megaflow lookup code in modification path into
> > relying the ufid to find the correct megaflow instead of key lookup.
> >
> > Signed-off-by: Peng He
> > ---
>
> Thanks for the update! This version looks logically correct to me.
> See some small comments below.
>
> Best regards, Ilya Maximets.
>
> > lib/dpif-netdev.c | 49 ---
> > tests/pmd.at | 48 ++
> > 2 files changed, 82 insertions(+), 15 deletions(-)
> >
> > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> > index 70b953ae6..8c88a5dc0 100644
> > --- a/lib/dpif-netdev.c
> > +++ b/lib/dpif-netdev.c
> > @@ -4191,7 +4191,7 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
> > const struct dpif_flow_put *put,
> > struct dpif_flow_stats *stats)
> > {
> > -struct dp_netdev_flow *netdev_flow;
> > +struct dp_netdev_flow *netdev_flow = NULL;
> > int error = 0;
> >
> > if (stats) {
> > @@ -4199,16 +4199,39 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
> > }
> >
> > ovs_mutex_lock(>flow_mutex);
> > -netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > -if (!netdev_flow) {
> > -if (put->flags & DPIF_FP_CREATE) {
> > -dp_netdev_flow_add(pmd, match, ufid, put->actions,
> > - put->actions_len, ODPP_NONE);
> > +if (put->ufid) {
> > +netdev_flow = dp_netdev_pmd_find_flow(pmd, put->ufid,
> > + put->key, put->key_len);
> > +} else {
> > +/* Use key instead of the locally generated ufid
> > + * to search netdev_flow.
> > + */
>
> Nit: Usually, /* and */ in the comment are on their own lines or
> both on the lines with the text, i.e.
>
> /*
> * Line 1
> * Line 2
> */
>
> Or
>
> /* Line 1
> * Line 2 */
>
> Rarely the style is mixed. In the case above, I'd suggest to move
> the closing '*/' to the previous line.
>
> > +netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > +}
> > +
> > +if (put->flags & DPIF_FP_CREATE) {
> > +if (!netdev_flow) {
> > +/* If ufid is provided, use provided ufid, otherwise
> > + * use locally generated ufid.
> > + */
> > +dp_netdev_flow_add(pmd, match, put->ufid ? put->ufid : ufid,
>
> This looks abit strange, because ufid equals put->ufid, if provided.
> I'd remove the comment and just use 'ufid' here without a ternary
> operator.
>
> Or maybe a cleaner solution is to remove 'ufid = *put->ufid;' assignment
> from dpif_netdev_flow_put() and re-name the argument of flow_put_on_pmd()
> function from 'ufid' to 'local_ufid' or something like that. In this
> case we should still remove the comment, but keep the ternary operator.
> Might be easier to read this way. What do you think?
>
I miss ufid = put->uf
Re: [ovs-dev] [ovs-dev v3] dpif-netdev: fix dpif_netdev_flow_put
Hi, Ilya,
are there any more comments beside the Fixes tag?
Peng He 于2023年7月31日周一 20:53写道：
> just found that I miss the Fixes tag, will add it in the next post.
>
> Peng He 于2023年7月31日周一 14:44写道：
>
>> OVS allows overlapping megaflows, as long as the actions of these
>> megaflows are equal. However, the current implementation of action
>> modification relies on flow_lookup instead of ufid, this could result
>> in looking up a wrong megaflow and make the ukeys and megaflows
>> inconsistent
>>
>> Just like the test case in the patch, at first we have a rule with the
>> prefix:
>>
>> 10.1.2.0/24
>>
>> and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with
>> IP
>> 10.1.2.2 is received.
>>
>> Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep the
>> 10.1.2.2/24 megaflow and just changes its action instead of extending
>> the prefix into 10.1.2.2/16.
>>
>> then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
>> this time, we will have an overlapping megaflow with the right prefix:
>> 10.1.0.2/16
>>
>> now we have two megaflows:
>> 10.1.2.2/24
>> 10.1.0.2/16
>>
>> last, suppose we have changed the ruleset again. The revalidator this
>> time still decides to change the actions of both megaflows instead of
>> deleting them.
>>
>> The dpif_netdev_flow_put will search the megaflow to modify with unmasked
>> keys, however it might lookup the wrong megaflow as the key 10.1.2.2
>> matches
>> both 10.1.2.2/24 and 10.1.0.2/16!
>>
>> This patch changes the megaflow lookup code in modification path into
>> relying the ufid to find the correct megaflow instead of key lookup.
>>
>> Signed-off-by: Peng He
>> ---
>> lib/dpif-netdev.c | 49 ---
>> tests/pmd.at | 48 ++
>> 2 files changed, 82 insertions(+), 15 deletions(-)
>>
>> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
>> index 70b953ae6..8c88a5dc0 100644
>> --- a/lib/dpif-netdev.c
>> +++ b/lib/dpif-netdev.c
>> @@ -4191,7 +4191,7 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
>> const struct dpif_flow_put *put,
>> struct dpif_flow_stats *stats)
>> {
>> -struct dp_netdev_flow *netdev_flow;
>> +struct dp_netdev_flow *netdev_flow = NULL;
>> int error = 0;
>>
>> if (stats) {
>> @@ -4199,16 +4199,39 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
>> }
>>
>> ovs_mutex_lock(>flow_mutex);
>> -netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
>> -if (!netdev_flow) {
>> -if (put->flags & DPIF_FP_CREATE) {
>> -dp_netdev_flow_add(pmd, match, ufid, put->actions,
>> - put->actions_len, ODPP_NONE);
>> +if (put->ufid) {
>> +netdev_flow = dp_netdev_pmd_find_flow(pmd, put->ufid,
>> + put->key, put->key_len);
>> +} else {
>> +/* Use key instead of the locally generated ufid
>> + * to search netdev_flow.
>> + */
>> +netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
>> +}
>> +
>> +if (put->flags & DPIF_FP_CREATE) {
>> +if (!netdev_flow) {
>> +/* If ufid is provided, use provided ufid, otherwise
>> + * use locally generated ufid.
>> + */
>> +dp_netdev_flow_add(pmd, match, put->ufid ? put->ufid : ufid,
>> + put->actions, put->actions_len,
>> ODPP_NONE);
>> } else {
>> -error = ENOENT;
>> +error = EEXIST;
>> }
>> -} else {
>> -if (put->flags & DPIF_FP_MODIFY) {
>> +goto exit;
>> +}
>> +
>> +if (put->flags & DPIF_FP_MODIFY) {
>> +if (!netdev_flow) {
>> +error = ENOENT;
>> +} else {
>> +if (!flow_equal(>flow, _flow->flow)) {
>> +/* Overlapping flow. */
>> +error = EINVAL;
>> +goto exit;
>> +}
>> +
>> struct dp_netdev_actions *new_actions;
>> struct dp_netdev_actions *old_actions;
>>
>> @@ -4239,15 +4262,11 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
>> * counter, and subtract
Re: [ovs-dev] [ovs-dev v3] dpif-netdev: fix dpif_netdev_flow_put
just found that I miss the Fixes tag, will add it in the next post.
Peng He 于2023年7月31日周一 14:44写道：
> OVS allows overlapping megaflows, as long as the actions of these
> megaflows are equal. However, the current implementation of action
> modification relies on flow_lookup instead of ufid, this could result
> in looking up a wrong megaflow and make the ukeys and megaflows
> inconsistent
>
> Just like the test case in the patch, at first we have a rule with the
> prefix:
>
> 10.1.2.0/24
>
> and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with IP
> 10.1.2.2 is received.
>
> Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep the
> 10.1.2.2/24 megaflow and just changes its action instead of extending
> the prefix into 10.1.2.2/16.
>
> then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
> this time, we will have an overlapping megaflow with the right prefix:
> 10.1.0.2/16
>
> now we have two megaflows:
> 10.1.2.2/24
> 10.1.0.2/16
>
> last, suppose we have changed the ruleset again. The revalidator this
> time still decides to change the actions of both megaflows instead of
> deleting them.
>
> The dpif_netdev_flow_put will search the megaflow to modify with unmasked
> keys, however it might lookup the wrong megaflow as the key 10.1.2.2
> matches
> both 10.1.2.2/24 and 10.1.0.2/16!
>
> This patch changes the megaflow lookup code in modification path into
> relying the ufid to find the correct megaflow instead of key lookup.
>
> Signed-off-by: Peng He
> ---
> lib/dpif-netdev.c | 49 ---
> tests/pmd.at | 48 ++
> 2 files changed, 82 insertions(+), 15 deletions(-)
>
> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> index 70b953ae6..8c88a5dc0 100644
> --- a/lib/dpif-netdev.c
> +++ b/lib/dpif-netdev.c
> @@ -4191,7 +4191,7 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
> const struct dpif_flow_put *put,
> struct dpif_flow_stats *stats)
> {
> -struct dp_netdev_flow *netdev_flow;
> +struct dp_netdev_flow *netdev_flow = NULL;
> int error = 0;
>
> if (stats) {
> @@ -4199,16 +4199,39 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
> }
>
> ovs_mutex_lock(>flow_mutex);
> -netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> -if (!netdev_flow) {
> -if (put->flags & DPIF_FP_CREATE) {
> -dp_netdev_flow_add(pmd, match, ufid, put->actions,
> - put->actions_len, ODPP_NONE);
> +if (put->ufid) {
> +netdev_flow = dp_netdev_pmd_find_flow(pmd, put->ufid,
> + put->key, put->key_len);
> +} else {
> +/* Use key instead of the locally generated ufid
> + * to search netdev_flow.
> + */
> +netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> +}
> +
> +if (put->flags & DPIF_FP_CREATE) {
> +if (!netdev_flow) {
> +/* If ufid is provided, use provided ufid, otherwise
> + * use locally generated ufid.
> + */
> +dp_netdev_flow_add(pmd, match, put->ufid ? put->ufid : ufid,
> + put->actions, put->actions_len, ODPP_NONE);
> } else {
> -error = ENOENT;
> +error = EEXIST;
> }
> -} else {
> -if (put->flags & DPIF_FP_MODIFY) {
> +goto exit;
> +}
> +
> +if (put->flags & DPIF_FP_MODIFY) {
> +if (!netdev_flow) {
> +error = ENOENT;
> +} else {
> +if (!flow_equal(>flow, _flow->flow)) {
> +/* Overlapping flow. */
> +error = EINVAL;
> +goto exit;
> +}
> +
> struct dp_netdev_actions *new_actions;
> struct dp_netdev_actions *old_actions;
>
> @@ -4239,15 +4262,11 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
> * counter, and subtracting it before outputting the
> stats */
> error = EOPNOTSUPP;
> }
> -
> ovsrcu_postpone(dp_netdev_actions_free, old_actions);
> -} else if (put->flags & DPIF_FP_CREATE) {
> -error = EEXIST;
> -} else {
> -/* Overlapping flow. */
> -error = EINVAL;
> }
> }
> +
> +exit:
> ovs_mutex_unlock(>flow_mutex);
> return error;
> }
> diff --git a/tests/pmd.at b/tests/pmd.at
>
[ovs-dev] [ovs-dev v3] dpif-netdev: fix dpif_netdev_flow_put
OVS allows overlapping megaflows, as long as the actions of these
megaflows are equal. However, the current implementation of action
modification relies on flow_lookup instead of ufid, this could result
in looking up a wrong megaflow and make the ukeys and megaflows inconsistent
Just like the test case in the patch, at first we have a rule with the
prefix:
10.1.2.0/24
and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with IP
10.1.2.2 is received.
Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep the
10.1.2.2/24 megaflow and just changes its action instead of extending
the prefix into 10.1.2.2/16.
then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
this time, we will have an overlapping megaflow with the right prefix:
10.1.0.2/16
now we have two megaflows:
10.1.2.2/24
10.1.0.2/16
last, suppose we have changed the ruleset again. The revalidator this
time still decides to change the actions of both megaflows instead of
deleting them.
The dpif_netdev_flow_put will search the megaflow to modify with unmasked
keys, however it might lookup the wrong megaflow as the key 10.1.2.2 matches
both 10.1.2.2/24 and 10.1.0.2/16!
This patch changes the megaflow lookup code in modification path into
relying the ufid to find the correct megaflow instead of key lookup.
Signed-off-by: Peng He
---
lib/dpif-netdev.c | 49 ---
tests/pmd.at | 48 ++
2 files changed, 82 insertions(+), 15 deletions(-)
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index 70b953ae6..8c88a5dc0 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -4191,7 +4191,7 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
const struct dpif_flow_put *put,
struct dpif_flow_stats *stats)
{
-struct dp_netdev_flow *netdev_flow;
+struct dp_netdev_flow *netdev_flow = NULL;
int error = 0;
if (stats) {
@@ -4199,16 +4199,39 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
}
ovs_mutex_lock(>flow_mutex);
-netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
-if (!netdev_flow) {
-if (put->flags & DPIF_FP_CREATE) {
-dp_netdev_flow_add(pmd, match, ufid, put->actions,
- put->actions_len, ODPP_NONE);
+if (put->ufid) {
+netdev_flow = dp_netdev_pmd_find_flow(pmd, put->ufid,
+ put->key, put->key_len);
+} else {
+/* Use key instead of the locally generated ufid
+ * to search netdev_flow.
+ */
+netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
+}
+
+if (put->flags & DPIF_FP_CREATE) {
+if (!netdev_flow) {
+/* If ufid is provided, use provided ufid, otherwise
+ * use locally generated ufid.
+ */
+dp_netdev_flow_add(pmd, match, put->ufid ? put->ufid : ufid,
+ put->actions, put->actions_len, ODPP_NONE);
} else {
-error = ENOENT;
+error = EEXIST;
}
-} else {
-if (put->flags & DPIF_FP_MODIFY) {
+goto exit;
+}
+
+if (put->flags & DPIF_FP_MODIFY) {
+if (!netdev_flow) {
+error = ENOENT;
+} else {
+if (!flow_equal(>flow, _flow->flow)) {
+/* Overlapping flow. */
+error = EINVAL;
+goto exit;
+}
+
struct dp_netdev_actions *new_actions;
struct dp_netdev_actions *old_actions;
@@ -4239,15 +4262,11 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
* counter, and subtracting it before outputting the stats */
error = EOPNOTSUPP;
}
-
ovsrcu_postpone(dp_netdev_actions_free, old_actions);
-} else if (put->flags & DPIF_FP_CREATE) {
-error = EEXIST;
-} else {
-/* Overlapping flow. */
-error = EINVAL;
}
}
+
+exit:
ovs_mutex_unlock(>flow_mutex);
return error;
}
diff --git a/tests/pmd.at b/tests/pmd.at
index 48f3d432d..f399294d2 100644
--- a/tests/pmd.at
+++ b/tests/pmd.at
@@ -1300,3 +1300,51 @@ OVS_WAIT_UNTIL([tail -n +$LINENUM ovs-vswitchd.log |
grep "PMD load based sleeps
OVS_VSWITCHD_STOP
AT_CLEANUP
+
+AT_SETUP([PMD - revalidator wrongly modify userspace megaflows])
+
+OVS_VSWITCHD_START(
+[add-port br0 p1 \
+ -- set interface p1 type=dummy-pmd \
+ -- set bridge br0 datapath-type=dummy \
+ -- add-port br0 p2 \
+ -- set interface p2 type=dummy-pmd --
+])
+
+dnl Add one openflow rule and generate a megaflow.
+AT_CHECK([ovs-ofctl add-flow br0
'table=0,in_port=p1,ip,nw_dst=10.1.2.0/24,actions=p2'])
+AT_CHECK([ovs-appctl netdev-dummy/receive p1
'ipv4(src=10.0.0.1,dst=10.1.2.2),tcp(src=1,ds
Re: [ovs-dev] [ovs-dev v4] dpif-netdev: fix dpif_netdev_flow_put
Ilya Maximets 于2023年7月28日周五 20:25写道： > On 7/28/23 11:02, Peng He wrote: > > > > > > Ilya Maximets mailto:i.maxim...@ovn.org>> > 于2023年7月28日周五 00:59写道： > > > > On 7/27/23 04:31, Peng He wrote: > > > > > > > > > Eelco Chaudron mailto:echau...@redhat.com> > <mailto:echau...@redhat.com <mailto:echau...@redhat.com>>> 于2023年7月6日周四 > 15:52写道： > > > > > > > > > > > > On 6 Jul 2023, at 4:32, Peng He wrote: > > > > > > > Eelco Chaudron echau...@redhat.com> <mailto:echau...@redhat.com echau...@redhat.com>>> 于2023年7月5日周三 22:16写道： > > > > > > > >> > > > >> > > > >> On 1 Jul 2023, at 4:43, Peng He wrote: > > > >> > > > >>> OVS allows overlapping megaflows, as long as the actions > of these > > > >>> megaflows are equal. However, the current implementation > of action > > > >>> modification relies on flow_lookup instead of ufid, this > could result > > > >>> in looking up a wrong megaflow and make the ukeys and > megaflows > > > >> inconsistent > > > >>> > > > >>> Just like the test case in the patch, at first we have a > rule with the > > > >>> prefix: > > > >>> > > > >>> 10.1.2.0/24 <http://10.1.2.0/24> <http://10.1.2.0/24 < > http://10.1.2.0/24>> > > > >>> > > > >>> and we will get a megaflow with prefixes 10.1.2.2/24 < > http://10.1.2.2/24> <http://10.1.2.2/24 <http://10.1.2.2/24>> when a > packet with > > > >> IP > > > >>> 10.1.2.2 is received. > > > >>> > > > >>> Then suppose we change the rule into 10.1.0.0/16 < > http://10.1.0.0/16> <http://10.1.0.0/16 <http://10.1.0.0/16>>. OVS > prefers to keep > > > >> the > > > >>> 10.1.2.2/24 <http://10.1.2.2/24> <http://10.1.2.2/24 < > http://10.1.2.2/24>> megaflow and just changes its action instead of > extending > > > >>> the prefix into 10.1.2.2/16 <http://10.1.2.2/16> < > http://10.1.2.2/16 <http://10.1.2.2/16>>. > > > >>> > > > >>> then suppose we have a 10.1.0.2 packet, since it misses > the megaflow, > > > >>> this time, we will have an overlapping megaflow with the > right prefix: > > > >>> 10.1.0.2/16 <http://10.1.0.2/16> <http://10.1.0.2/16 < > http://10.1.0.2/16>> > > > >>> > > > >>> now we have two megaflows: > > > >>> 10.1.2.2/24 <http://10.1.2.2/24> <http://10.1.2.2/24 < > http://10.1.2.2/24>> > > > >>> 10.1.0.2/16 <http://10.1.0.2/16> <http://10.1.0.2/16 < > http://10.1.0.2/16>> > > > >>> > > > >>> last, suppose we have changed the ruleset again. The > revalidator this > > > >>> time still decides to change the actions of both megaflows > instead of > > > >>> deleting them. > > > >>> > > > >>> The dpif_netdev_flow_put will search the megaflow to > modify with unmasked > > > >>> keys, however it might lookup the wrong megaflow as the > key 10.1.2.2 > > > >> matches > > > >>> both 10.1.2.2/24 <http://10.1.2.2/24> <http://10.1.2.2/24 > <http://10.1.2.2/24>> and 10.1.0.2/16 <http://10.1.0.2/16> < > http://10.1.0.2/16 <http://10.1.0.2/16>>! > > > >>> > > > >>> This patch changes the megaflow lookup code in > modification path into > > > >>> relying the ufid to find the correct megaflow instead of > key lookup. > > > >> > > > >> Thanks for fixing Ilya’s comments! I’ve also copied in some > of the v3 > > > >> discussion, so we can wrap it up here. > > > >> > > > >> Acked-by: Eelco Chaudron echau...@redhat.com> <mailto:echau...@redhat.com echau...@redhat.com>>> > &
Re: [ovs-dev] [ovs-dev v4] dpif-netdev: fix dpif_netdev_flow_put
Ilya Maximets 于2023年7月28日周五 00:59写道：
> On 7/27/23 04:31, Peng He wrote:
> >
> >
> > Eelco Chaudron mailto:echau...@redhat.com>>
> 于2023年7月6日周四 15:52写道：
> >
> >
> >
> > On 6 Jul 2023, at 4:32, Peng He wrote:
> >
> > > Eelco Chaudron mailto:echau...@redhat.com>>
> 于2023年7月5日周三 22:16写道：
> > >
> > >>
> > >>
> > >> On 1 Jul 2023, at 4:43, Peng He wrote:
> > >>
> > >>> OVS allows overlapping megaflows, as long as the actions of these
> > >>> megaflows are equal. However, the current implementation of
> action
> > >>> modification relies on flow_lookup instead of ufid, this could
> result
> > >>> in looking up a wrong megaflow and make the ukeys and megaflows
> > >> inconsistent
> > >>>
> > >>> Just like the test case in the patch, at first we have a rule
> with the
> > >>> prefix:
> > >>>
> > >>> 10.1.2.0/24 <http://10.1.2.0/24>
> > >>>
> > >>> and we will get a megaflow with prefixes 10.1.2.2/24 <
> http://10.1.2.2/24> when a packet with
> > >> IP
> > >>> 10.1.2.2 is received.
> > >>>
> > >>> Then suppose we change the rule into 10.1.0.0/16 <
> http://10.1.0.0/16>. OVS prefers to keep
> > >> the
> > >>> 10.1.2.2/24 <http://10.1.2.2/24> megaflow and just changes its
> action instead of extending
> > >>> the prefix into 10.1.2.2/16 <http://10.1.2.2/16>.
> > >>>
> > >>> then suppose we have a 10.1.0.2 packet, since it misses the
> megaflow,
> > >>> this time, we will have an overlapping megaflow with the right
> prefix:
> > >>> 10.1.0.2/16 <http://10.1.0.2/16>
> > >>>
> > >>> now we have two megaflows:
> > >>> 10.1.2.2/24 <http://10.1.2.2/24>
> > >>> 10.1.0.2/16 <http://10.1.0.2/16>
> > >>>
> > >>> last, suppose we have changed the ruleset again. The revalidator
> this
> > >>> time still decides to change the actions of both megaflows
> instead of
> > >>> deleting them.
> > >>>
> > >>> The dpif_netdev_flow_put will search the megaflow to modify with
> unmasked
> > >>> keys, however it might lookup the wrong megaflow as the key
> 10.1.2.2
> > >> matches
> > >>> both 10.1.2.2/24 <http://10.1.2.2/24> and 10.1.0.2/16 <
> http://10.1.0.2/16>!
> > >>>
> > >>> This patch changes the megaflow lookup code in modification path
> into
> > >>> relying the ufid to find the correct megaflow instead of key
> lookup.
> > >>
> > >> Thanks for fixing Ilya’s comments! I’ve also copied in some of
> the v3
> > >> discussion, so we can wrap it up here.
> > >>
> > >> Acked-by: Eelco Chaudron echau...@redhat.com>>
> > >>
> > >> //Eelco
> > >>
> > >>> Signed-off-by: Peng He hepeng.0...@bytedance.com>>
> > >>> ---
> > >>> lib/dpif-netdev.c | 62
> ++-
> > >>> tests/pmd.at <http://pmd.at> | 48
>
> > >>> 2 files changed, 82 insertions(+), 28 deletions(-)
> > >>>
> > >>> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> > >>> index 70b953ae6..b90ed1542 100644
> > >>> --- a/lib/dpif-netdev.c
> > >>> +++ b/lib/dpif-netdev.c
> > >>> @@ -4198,36 +4198,43 @@ flow_put_on_pmd(struct
> dp_netdev_pmd_thread *pmd,
> > >>> memset(stats, 0, sizeof *stats);
> > >>> }
> > >>>
> > >>> -ovs_mutex_lock(>flow_mutex);
> > >>> -netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > >>> -if (!netdev_flow) {
> > >>> -if (put->flags & DPIF_FP_CREATE) {
> > >>> +if (put->flags & DPIF_FP_CREATE) {
> > >>
> > >>
> > >> EC> Should this not be:
> > >> EC>
Re: [ovs-dev] [ovs-dev v4] dpif-netdev: fix dpif_netdev_flow_put
Eelco Chaudron 于2023年7月6日周四 15:52写道：
>
>
> On 6 Jul 2023, at 4:32, Peng He wrote:
>
> > Eelco Chaudron 于2023年7月5日周三 22:16写道：
> >
> >>
> >>
> >> On 1 Jul 2023, at 4:43, Peng He wrote:
> >>
> >>> OVS allows overlapping megaflows, as long as the actions of these
> >>> megaflows are equal. However, the current implementation of action
> >>> modification relies on flow_lookup instead of ufid, this could result
> >>> in looking up a wrong megaflow and make the ukeys and megaflows
> >> inconsistent
> >>>
> >>> Just like the test case in the patch, at first we have a rule with the
> >>> prefix:
> >>>
> >>> 10.1.2.0/24
> >>>
> >>> and we will get a megaflow with prefixes 10.1.2.2/24 when a packet
> with
> >> IP
> >>> 10.1.2.2 is received.
> >>>
> >>> Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep
> >> the
> >>> 10.1.2.2/24 megaflow and just changes its action instead of extending
> >>> the prefix into 10.1.2.2/16.
> >>>
> >>> then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
> >>> this time, we will have an overlapping megaflow with the right prefix:
> >>> 10.1.0.2/16
> >>>
> >>> now we have two megaflows:
> >>> 10.1.2.2/24
> >>> 10.1.0.2/16
> >>>
> >>> last, suppose we have changed the ruleset again. The revalidator this
> >>> time still decides to change the actions of both megaflows instead of
> >>> deleting them.
> >>>
> >>> The dpif_netdev_flow_put will search the megaflow to modify with
> unmasked
> >>> keys, however it might lookup the wrong megaflow as the key 10.1.2.2
> >> matches
> >>> both 10.1.2.2/24 and 10.1.0.2/16!
> >>>
> >>> This patch changes the megaflow lookup code in modification path into
> >>> relying the ufid to find the correct megaflow instead of key lookup.
> >>
> >> Thanks for fixing Ilya’s comments! I’ve also copied in some of the v3
> >> discussion, so we can wrap it up here.
> >>
> >> Acked-by: Eelco Chaudron
> >>
> >> //Eelco
> >>
> >>> Signed-off-by: Peng He
> >>> ---
> >>> lib/dpif-netdev.c | 62 ++-
> >>> tests/pmd.at | 48
> >>> 2 files changed, 82 insertions(+), 28 deletions(-)
> >>>
> >>> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> >>> index 70b953ae6..b90ed1542 100644
> >>> --- a/lib/dpif-netdev.c
> >>> +++ b/lib/dpif-netdev.c
> >>> @@ -4198,36 +4198,43 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread
> *pmd,
> >>> memset(stats, 0, sizeof *stats);
> >>> }
> >>>
> >>> -ovs_mutex_lock(>flow_mutex);
> >>> -netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> >>> -if (!netdev_flow) {
> >>> -if (put->flags & DPIF_FP_CREATE) {
> >>> +if (put->flags & DPIF_FP_CREATE) {
> >>
> >>
> >> EC> Should this not be:
> >> EC>
> >> EC> if (put->flags & DPIF_FP_CREATE && !(put->flags & DPIF_FP_MODIFY))
> >> EC>
> >> EC> I know this will break the fix, but I’m wondering what the possible
> >> EC> combinations are.
> >> EC> Quickly looking at the code the following ones seem to exist:
> >> EC>
> >> EC> DPIF_FP_CREATE -> Create a flow, if one exists return EEXIST
> >> EC> DPIF_FP_MODIFY -> Modify a flow, if it does not exist return ENONT
> >> EC> DPIF_FP_CREATE | DPIF_FP_MODIFY -> If it exists modify it, if it
> does
> >> EC> not exists create it.
> >> EC>
> >> EC> Could the last combination not potentially fail the lookup?
> >> EC>
> >> EC> Or are we assuming only standalone DPIF_FP_MODIFY’s are the
> problem? In
> >> EC> theory, it could also be the combination.
> >> EC>
> >>
> >> PH> sorry, I was wrong. Such a combination does exist in
> >> PH>the dpif_probe_feature, and it probes the datapath by trying to put
> >> flows:
> >> PH>
> >> PH> error = dpif_flow_put(dpif, DPIF_FP_CREATE | DPIF_FP_MODIFY |
> >> PH>DPIF_FP_PROBE,
> &
Re: [ovs-dev] [ovs-dev v12] ofproto-dpif-upcall: fix push_dp_ops
Simon Horman 于2023年7月27日周四 01:04写道： > On Thu, Jul 06, 2023 at 04:59:36PM +0800, Peng He wrote: > > ... > > > >>>> +dnl Replace OpenFlow rules, trigger revalidation and wait for it to > > >>> complete. > > >>>> +AT_CHECK([echo 'table=0,in_port=p1,ip actions=ct(commit)' | > ovs-ofctl > > >>> --bundle replace-flows br0 -]) > > >>>> +AT_CHECK([ovs-appctl revalidator/wait]) > > >>>> + > > >>>> +dnl Inconsistent ukey should be deleted. > > >>>> +AT_CHECK([ovs-appctl upcall/show | grep keys | grep -q -v 0], [1]) > > >>>> + > > >>>> +dnl Check the log for the flow modification error. > > >>>> +AT_CHECK([grep -q -E ".*failed to put.*$" ovs-vswitchd.log]) > > >>>> + > > >>>> +dnl Remove warning logs to let test suite pass. > > >>>> +OVS_VSWITCHD_STOP(["dnl > > >>>> +/.*failed to put.*$/d > > >>>> +/.*failed to flow_del.*$/d"]) > > >>> > > >>> You missed the indentation suggested by Ilya: > > >>> > > >>> > > >> Which kind of the email client are you using... > > >> The indentation here is really easy to miss > > > > > > From the User-Agent data, he seems to be using Thunderbird/102.10.0 > but the emails are plain text, so the problem is probably with your email > client converting them to “html” like format. If you can configure a fixed > width font for text emails you might spot these thinks easier. > > > > > > I change to use another email client now, thanks ! > > Hi Peng He, all, > > Am I correct in assuming that there will be a v13? > Hi, if need, I can submit a v13, the current version is just missing some indentation. "The changes look good, with one little nit (see below), but those can be applied during commit." I thought it would be changed by the maintainer during merging... -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] The file-column-diff might lead to break db file
Hi, Ilya,
After applying the patch, the test suite has passed.
Looking forward to your complete fix.
Ilya Maximets 于2023年7月21日周五 21:09写道：
> On 7/21/23 14:44, Peng He wrote:
> > Hi, Ilya,
> >
> > Suppose you have a column which is a set with limited integers (suppose
> min: 0, max: 2), the following operation will generate a record in db file
> with 3 integers:
> >
> > first, set column A to [0],
> > then, set column A to [1,2],
> >
> > after that the column A is recorded as [0,1,2], as [0] diff [1,2] == [0,
> 1, 2].
>
> Hmm, good catch!
>
> The file is correct. However, the ovsdb-server is too strict while reading
> these records. Could you try the following change:
>
> diff --git a/ovsdb/file.c b/ovsdb/file.c
> index 400b34794..6e7e47ca6 100644
> --- a/ovsdb/file.c
> +++ b/ovsdb/file.c
> @@ -107,7 +107,14 @@ ovsdb_file_update_row_from_json(struct ovsdb_row
> *row, bool converting,
>column_name, schema->name);
> }
>
> -error = ovsdb_datum_from_json(, >type, node->data,
> NULL);
> +if (row_contains_diff) {
> +/* Diff may violate the type size rules. */
> +error = ovsdb_transient_datum_from_json(, >type,
> +node->data);
> +} else {
> +error = ovsdb_datum_from_json(, >type,
> + node->data, NULL);
> +}
> if (error) {
> return error;
> }
> ---
>
> ?
>
> The type constraints will be evaluated later while applying the diff
> to the existing row.
>
> Best regards, Ilya Maximets.
>
> > Such a bug can be reproduced by a test suite below:
> >
> >
> > AT_SETUP([ovsdb-server diff columns])
> > on_exit 'kill `cat *.pid`'
> > AT_DATA([schema],
> > [[{"name": "mydb",
> > "tables": {
> >"Root": {
> > "columns": {
> >"managers": {
> > "type": {
> >"key": {"type": "uuid", "refTable": "Manager"},
> >"min": 0,
> >"max": "unlimited",
> >"Manager": {
> > "columns": {
> >"target": {
> > "type": "string"},
> >"read_only": {
> > "type": {
> >"key": "boolean",
> >"min": 0,
> >"max": 1}},
> >"is_connected": {
> > "type": {
> >"key": "boolean",
> >"min": 0,
> >"max": 1,
> >"ordinals": {
> > "columns": {
> >"number": {"type": {
> > "key": {"type" : "integer"},
> > "min": 0,
> > "max": 2}},
> >"name": {"type": "string"}},
> > "indexes": [["name"]]}
> > },
> > "version": "5.1.3",
> > "cksum": "12345678 9"
> > }
> > ]])
> > AT_CHECK([ovsdb-tool create db schema], [0], [ignore], [ignore])
> > AT_CHECK(
> > [[ovsdb-tool transact db \
> > '["mydb",
> >{"op": "insert",
> > "table": "Root",
> > "row": {
> > "managers": ["set", [["named-uuid", "x"]]]}},
> >{"op": "insert",
> > "table": "Manager",
> > "uuid-name": "x",
> > "row": {"target": "ptcp:0:127.0.0.1",
> >"read_only": false}}]']], [0], [ignore], [ignore])
> >
> > AT_CHECK([ovsdb-server --detach --no-chdir --log-file --pidfile
> --remote=db:mydb,Root,managers db], [0], [ignore], [ignore])
> > PARSE_LISTENING_PORT([ovsdb-server.log], [TCP_PORT])
> > AT_CHECK([ovsdb-client get-schema-version tcp:127.0.0.1:$TCP_PORT
> mydb], [0], [5.1.3
> > ])
> >
> > AT_CHECK([ovsdb-client transact tcp:127.0.0.
[ovs-dev] The file-column-diff might lead to break db file
Hi, Ilya,
Suppose you have a column which is a set with limited integers (suppose
min: 0, max: 2), the following operation will generate a record in db file
with 3 integers:
first, set column A to [0],
then, set column A to [1,2],
after that the column A is recorded as [0,1,2], as [0] diff [1,2] == [0, 1,
2].
Such a bug can be reproduced by a test suite below:
AT_SETUP([ovsdb-server diff columns])
on_exit 'kill `cat *.pid`'
AT_DATA([schema],
[[{"name": "mydb",
"tables": {
"Root": {
"columns": {
"managers": {
"type": {
"key": {"type": "uuid", "refTable": "Manager"},
"min": 0,
"max": "unlimited",
"Manager": {
"columns": {
"target": {
"type": "string"},
"read_only": {
"type": {
"key": "boolean",
"min": 0,
"max": 1}},
"is_connected": {
"type": {
"key": "boolean",
"min": 0,
"max": 1,
"ordinals": {
"columns": {
"number": {"type": {
"key": {"type" : "integer"},
"min": 0,
"max": 2}},
"name": {"type": "string"}},
"indexes": [["name"]]}
},
"version": "5.1.3",
"cksum": "12345678 9"
}
]])
AT_CHECK([ovsdb-tool create db schema], [0], [ignore], [ignore])
AT_CHECK(
[[ovsdb-tool transact db \
'["mydb",
{"op": "insert",
"table": "Root",
"row": {
"managers": ["set", [["named-uuid", "x"]]]}},
{"op": "insert",
"table": "Manager",
"uuid-name": "x",
"row": {"target": "ptcp:0:127.0.0.1",
"read_only": false}}]']], [0], [ignore], [ignore])
AT_CHECK([ovsdb-server --detach --no-chdir --log-file --pidfile
--remote=db:mydb,Root,managers db], [0], [ignore], [ignore])
PARSE_LISTENING_PORT([ovsdb-server.log], [TCP_PORT])
AT_CHECK([ovsdb-client get-schema-version tcp:127.0.0.1:$TCP_PORT mydb],
[0], [5.1.3
])
AT_CHECK([ovsdb-client transact tcp:127.0.0.1:$TCP_PORT \
['["mydb",
{"op": "insert",
"table": "ordinals",
"row": {"name": "two", "number": 0}}
]']], [0], [stdout], [ignore])
AT_CHECK([ovsdb-client transact tcp:127.0.0.1:$TCP_PORT \
['["mydb",
{"op": "update",
"table": "ordinals",
"where": [["name", "==", "two"]],
"row": {"number": ["set", [1,2]]}}
]']], [0], [stdout], [ignore])
AT_CHECK([ovs-appctl -t ovsdb-server exit])
AT_CHECK([ovsdb-server --detach --no-chdir --log-file --pidfile
--remote=db:mydb,Root,managers db], [0], [ignore], [ignore])
OVSDB_SERVER_SHUTDOWN(["
/No status column present in the Manager table/d
"])
AT_CLEANUP
when reboot ovsdb-server, it will warn in the log:
2023-07-21T12:43:55Z|1|vlog|INFO|opened log file
/root/ovs/tests/testsuite.dir/2044/ovsdb-server.log
2023-07-21T12:43:55Z|2|ovsdb_server|ERR|syntax "["set",[0,1,2]]":
syntax error: set must have 0 to 2 members but 3 are present
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v12] ofproto-dpif-upcall: fix push_dp_ops
> 2023年7月6日 15:47，Eelco Chaudron 写道：
>
>
>
> On 6 Jul 2023, at 4:33, Peng He wrote:
>
>> Eelco Chaudron 于2023年7月5日周三 21:24写道：
>>
>>>
>>>
>>> On 1 Jul 2023, at 7:11, Peng He wrote:
>>>
>>>> push_dp_ops only handles delete ops errors but ignores the modify
>>>> ops results. It's better to handle all the dp operation errors in
>>>> a consistent way.
>>>>
>>>> This patch prevents the inconsistency by considering modify failure
>>>> in revalidators.
>>>>
>>>> To note, we cannot perform two state transitions and change ukey_state
>>>> into UKEY_EVICTED directly here, because, if we do so, the
>>>> sweep will remove the ukey alone and leave dp flow alive. Later, the
>>>> dump will retrieve the dp flow and might even recover it. This will
>>>> contribute the stats of this dp flow twice.
>>>>
>>>> v8->v9: add testsuite and delete INCONSISTENT ukey at revalidate_sweep.
>>>> v9->v10: change the commit message and refine the test case.
>>>> v10->v11: fix indentation and refine the test case.
>>>> v11->v12: fix the test suite.
>>>>
>>>> Signed-off-by: Peng He
>>>
>>> Thanks for fixing the comments on v11. The changes look good, with one
>>> little nit (see below), but those can be applied during commit.
>>>
>>> Acked-by: Eelco Chaudron
>>>
>>>> ---
>>>> ofproto/ofproto-dpif-upcall.c | 50 +--
>>>> tests/dpif-netdev.at | 43 ++
>>>> 2 files changed, 79 insertions(+), 14 deletions(-)
>>>>
>>>> diff --git a/ofproto/ofproto-dpif-upcall.c
>>> b/ofproto/ofproto-dpif-upcall.c
>>>> index 04b583f81..cde03abc6 100644
>>>> --- a/ofproto/ofproto-dpif-upcall.c
>>>> +++ b/ofproto/ofproto-dpif-upcall.c
>>>> @@ -53,6 +53,7 @@
>>>> VLOG_DEFINE_THIS_MODULE(ofproto_dpif_upcall);
>>>>
>>>> COVERAGE_DEFINE(dumped_duplicate_flow);
>>>> +COVERAGE_DEFINE(dumped_inconsistent_flow);
>>>> COVERAGE_DEFINE(dumped_new_flow);
>>>> COVERAGE_DEFINE(handler_duplicate_upcall);
>>>> COVERAGE_DEFINE(revalidate_missed_dp_flow);
>>>> @@ -258,6 +259,7 @@ enum ukey_state {
>>>> UKEY_CREATED = 0,
>>>> UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is
>>> queued. */
>>>> UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed.
>>> */
>>>> +UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is
>>> inconsistent. */
>>>> UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is
>>> queued. */
>>>> UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
>>>> UKEY_DELETED, /* Ukey removed from umap, ukey free is
>>> deferred. */
>>>> @@ -1999,6 +2001,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
>>> ukey_state dst,
>>>> * UKEY_VISIBLE -> UKEY_EVICTED
>>>> * A handler attempts to install the flow, but the datapath
>>> rejects it.
>>>> * Consider that the datapath has already destroyed it.
>>>> + * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
>>>> + * A revalidator modifies the flow with error returns.
>>>> + * UKEY_INCONSISTENT -> UKEY_EVICTING
>>>> + * A revalidator decides to evict the datapath flow.
>>>> * UKEY_OPERATIONAL -> UKEY_EVICTING
>>>> * A revalidator decides to evict the datapath flow.
>>>> * UKEY_EVICTING-> UKEY_EVICTED
>>>> @@ -2006,8 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
>>> ukey_state dst,
>>>> * UKEY_EVICTED -> UKEY_DELETED
>>>> * A revalidator has removed the ukey from the umap and is
>>> deleting it.
>>>> */
>>>> -if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
>>>> - dst < UKEY_DELETED)) {
>>>> +if (ukey->state == dst - 1 ||
>>>> + (ukey->state == UKEY_VISIBLE && dst < UKEY_DELETED) ||
>>>> + (ukey->state == UKEY_OPERATIONAL && dst == UKEY_EVICTING)) {
>>>> ukey->state = dst;
>>>> } else {
>>>
Re: [ovs-dev] [ovs-dev v4] dpif-netdev: fix dpif_netdev_flow_put
Eelco Chaudron 于2023年7月6日周四 15:52写道：
>
>
> On 6 Jul 2023, at 4:32, Peng He wrote:
>
> > Eelco Chaudron 于2023年7月5日周三 22:16写道：
> >
> >>
> >>
> >> On 1 Jul 2023, at 4:43, Peng He wrote:
> >>
> >>> OVS allows overlapping megaflows, as long as the actions of these
> >>> megaflows are equal. However, the current implementation of action
> >>> modification relies on flow_lookup instead of ufid, this could result
> >>> in looking up a wrong megaflow and make the ukeys and megaflows
> >> inconsistent
> >>>
> >>> Just like the test case in the patch, at first we have a rule with the
> >>> prefix:
> >>>
> >>> 10.1.2.0/24
> >>>
> >>> and we will get a megaflow with prefixes 10.1.2.2/24 when a packet
> with
> >> IP
> >>> 10.1.2.2 is received.
> >>>
> >>> Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep
> >> the
> >>> 10.1.2.2/24 megaflow and just changes its action instead of extending
> >>> the prefix into 10.1.2.2/16.
> >>>
> >>> then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
> >>> this time, we will have an overlapping megaflow with the right prefix:
> >>> 10.1.0.2/16
> >>>
> >>> now we have two megaflows:
> >>> 10.1.2.2/24
> >>> 10.1.0.2/16
> >>>
> >>> last, suppose we have changed the ruleset again. The revalidator this
> >>> time still decides to change the actions of both megaflows instead of
> >>> deleting them.
> >>>
> >>> The dpif_netdev_flow_put will search the megaflow to modify with
> unmasked
> >>> keys, however it might lookup the wrong megaflow as the key 10.1.2.2
> >> matches
> >>> both 10.1.2.2/24 and 10.1.0.2/16!
> >>>
> >>> This patch changes the megaflow lookup code in modification path into
> >>> relying the ufid to find the correct megaflow instead of key lookup.
> >>
> >> Thanks for fixing Ilya’s comments! I’ve also copied in some of the v3
> >> discussion, so we can wrap it up here.
> >>
> >> Acked-by: Eelco Chaudron
> >>
> >> //Eelco
> >>
> >>> Signed-off-by: Peng He
> >>> ---
> >>> lib/dpif-netdev.c | 62 ++-
> >>> tests/pmd.at | 48
> >>> 2 files changed, 82 insertions(+), 28 deletions(-)
> >>>
> >>> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> >>> index 70b953ae6..b90ed1542 100644
> >>> --- a/lib/dpif-netdev.c
> >>> +++ b/lib/dpif-netdev.c
> >>> @@ -4198,36 +4198,43 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread
> *pmd,
> >>> memset(stats, 0, sizeof *stats);
> >>> }
> >>>
> >>> -ovs_mutex_lock(>flow_mutex);
> >>> -netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> >>> -if (!netdev_flow) {
> >>> -if (put->flags & DPIF_FP_CREATE) {
> >>> +if (put->flags & DPIF_FP_CREATE) {
> >>
> >>
> >> EC> Should this not be:
> >> EC>
> >> EC> if (put->flags & DPIF_FP_CREATE && !(put->flags & DPIF_FP_MODIFY))
> >> EC>
> >> EC> I know this will break the fix, but I’m wondering what the possible
> >> EC> combinations are.
> >> EC> Quickly looking at the code the following ones seem to exist:
> >> EC>
> >> EC> DPIF_FP_CREATE -> Create a flow, if one exists return EEXIST
> >> EC> DPIF_FP_MODIFY -> Modify a flow, if it does not exist return ENONT
> >> EC> DPIF_FP_CREATE | DPIF_FP_MODIFY -> If it exists modify it, if it
> does
> >> EC> not exists create it.
> >> EC>
> >> EC> Could the last combination not potentially fail the lookup?
> >> EC>
> >> EC> Or are we assuming only standalone DPIF_FP_MODIFY’s are the
> problem? In
> >> EC> theory, it could also be the combination.
> >> EC>
> >>
> >> PH> sorry, I was wrong. Such a combination does exist in
> >> PH>the dpif_probe_feature, and it probes the datapath by trying to put
> >> flows:
> >> PH>
> >> PH> error = dpif_flow_put(dpif, DPIF_FP_CREATE | DPIF_FP_MODIFY |
> >> PH>DPIF_FP_PROBE,
> >
Re: [ovs-dev] [ovs-dev v12] ofproto-dpif-upcall: fix push_dp_ops
Eelco Chaudron 于2023年7月5日周三 21:24写道：
>
>
> On 1 Jul 2023, at 7:11, Peng He wrote:
>
> > push_dp_ops only handles delete ops errors but ignores the modify
> > ops results. It's better to handle all the dp operation errors in
> > a consistent way.
> >
> > This patch prevents the inconsistency by considering modify failure
> > in revalidators.
> >
> > To note, we cannot perform two state transitions and change ukey_state
> > into UKEY_EVICTED directly here, because, if we do so, the
> > sweep will remove the ukey alone and leave dp flow alive. Later, the
> > dump will retrieve the dp flow and might even recover it. This will
> > contribute the stats of this dp flow twice.
> >
> > v8->v9: add testsuite and delete INCONSISTENT ukey at revalidate_sweep.
> > v9->v10: change the commit message and refine the test case.
> > v10->v11: fix indentation and refine the test case.
> > v11->v12: fix the test suite.
> >
> > Signed-off-by: Peng He
>
> Thanks for fixing the comments on v11. The changes look good, with one
> little nit (see below), but those can be applied during commit.
>
> Acked-by: Eelco Chaudron
>
> > ---
> > ofproto/ofproto-dpif-upcall.c | 50 +--
> > tests/dpif-netdev.at | 43 ++
> > 2 files changed, 79 insertions(+), 14 deletions(-)
> >
> > diff --git a/ofproto/ofproto-dpif-upcall.c
> b/ofproto/ofproto-dpif-upcall.c
> > index 04b583f81..cde03abc6 100644
> > --- a/ofproto/ofproto-dpif-upcall.c
> > +++ b/ofproto/ofproto-dpif-upcall.c
> > @@ -53,6 +53,7 @@
> > VLOG_DEFINE_THIS_MODULE(ofproto_dpif_upcall);
> >
> > COVERAGE_DEFINE(dumped_duplicate_flow);
> > +COVERAGE_DEFINE(dumped_inconsistent_flow);
> > COVERAGE_DEFINE(dumped_new_flow);
> > COVERAGE_DEFINE(handler_duplicate_upcall);
> > COVERAGE_DEFINE(revalidate_missed_dp_flow);
> > @@ -258,6 +259,7 @@ enum ukey_state {
> > UKEY_CREATED = 0,
> > UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is
> queued. */
> > UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed.
> */
> > +UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is
> inconsistent. */
> > UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is
> queued. */
> > UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
> > UKEY_DELETED, /* Ukey removed from umap, ukey free is
> deferred. */
> > @@ -1999,6 +2001,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
> ukey_state dst,
> > * UKEY_VISIBLE -> UKEY_EVICTED
> > * A handler attempts to install the flow, but the datapath
> rejects it.
> > * Consider that the datapath has already destroyed it.
> > + * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
> > + * A revalidator modifies the flow with error returns.
> > + * UKEY_INCONSISTENT -> UKEY_EVICTING
> > + * A revalidator decides to evict the datapath flow.
> > * UKEY_OPERATIONAL -> UKEY_EVICTING
> > * A revalidator decides to evict the datapath flow.
> > * UKEY_EVICTING-> UKEY_EVICTED
> > @@ -2006,8 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
> ukey_state dst,
> > * UKEY_EVICTED -> UKEY_DELETED
> > * A revalidator has removed the ukey from the umap and is
> deleting it.
> > */
> > -if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
> > - dst < UKEY_DELETED)) {
> > +if (ukey->state == dst - 1 ||
> > + (ukey->state == UKEY_VISIBLE && dst < UKEY_DELETED) ||
> > + (ukey->state == UKEY_OPERATIONAL && dst == UKEY_EVICTING)) {
> > ukey->state = dst;
> > } else {
> > struct ds ds = DS_EMPTY_INITIALIZER;
> > @@ -2490,26 +2497,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op
> *ops, size_t n_ops)
> >
> > for (i = 0; i < n_ops; i++) {
> > struct ukey_op *op = [i];
> > -struct dpif_flow_stats *push, *stats, push_buf;
> > -
> > -stats = op->dop.flow_del.stats;
> > -push = _buf;
> > -
> > -if (op->dop.type != DPIF_OP_FLOW_DEL) {
> > -/* Only deleted flows need their stats pushed. */
> > -continue;
> > -}
> >
> > if (op->dop.error) {
> > -/* flow_del error, 'stats' is unusable. */
> > if (op->uk
Re: [ovs-dev] [ovs-dev v4] dpif-netdev: fix dpif_netdev_flow_put
Eelco Chaudron 于2023年7月5日周三 22:16写道：
>
>
> On 1 Jul 2023, at 4:43, Peng He wrote:
>
> > OVS allows overlapping megaflows, as long as the actions of these
> > megaflows are equal. However, the current implementation of action
> > modification relies on flow_lookup instead of ufid, this could result
> > in looking up a wrong megaflow and make the ukeys and megaflows
> inconsistent
> >
> > Just like the test case in the patch, at first we have a rule with the
> > prefix:
> >
> > 10.1.2.0/24
> >
> > and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with
> IP
> > 10.1.2.2 is received.
> >
> > Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep
> the
> > 10.1.2.2/24 megaflow and just changes its action instead of extending
> > the prefix into 10.1.2.2/16.
> >
> > then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
> > this time, we will have an overlapping megaflow with the right prefix:
> > 10.1.0.2/16
> >
> > now we have two megaflows:
> > 10.1.2.2/24
> > 10.1.0.2/16
> >
> > last, suppose we have changed the ruleset again. The revalidator this
> > time still decides to change the actions of both megaflows instead of
> > deleting them.
> >
> > The dpif_netdev_flow_put will search the megaflow to modify with unmasked
> > keys, however it might lookup the wrong megaflow as the key 10.1.2.2
> matches
> > both 10.1.2.2/24 and 10.1.0.2/16!
> >
> > This patch changes the megaflow lookup code in modification path into
> > relying the ufid to find the correct megaflow instead of key lookup.
>
> Thanks for fixing Ilya’s comments! I’ve also copied in some of the v3
> discussion, so we can wrap it up here.
>
> Acked-by: Eelco Chaudron
>
> //Eelco
>
> > Signed-off-by: Peng He
> > ---
> > lib/dpif-netdev.c | 62 ++-
> > tests/pmd.at | 48
> > 2 files changed, 82 insertions(+), 28 deletions(-)
> >
> > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> > index 70b953ae6..b90ed1542 100644
> > --- a/lib/dpif-netdev.c
> > +++ b/lib/dpif-netdev.c
> > @@ -4198,36 +4198,43 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
> > memset(stats, 0, sizeof *stats);
> > }
> >
> > -ovs_mutex_lock(>flow_mutex);
> > -netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > -if (!netdev_flow) {
> > -if (put->flags & DPIF_FP_CREATE) {
> > +if (put->flags & DPIF_FP_CREATE) {
>
>
> EC> Should this not be:
> EC>
> EC> if (put->flags & DPIF_FP_CREATE && !(put->flags & DPIF_FP_MODIFY))
> EC>
> EC> I know this will break the fix, but I’m wondering what the possible
> EC> combinations are.
> EC> Quickly looking at the code the following ones seem to exist:
> EC>
> EC> DPIF_FP_CREATE -> Create a flow, if one exists return EEXIST
> EC> DPIF_FP_MODIFY -> Modify a flow, if it does not exist return ENONT
> EC> DPIF_FP_CREATE | DPIF_FP_MODIFY -> If it exists modify it, if it does
> EC> not exists create it.
> EC>
> EC> Could the last combination not potentially fail the lookup?
> EC>
> EC> Or are we assuming only standalone DPIF_FP_MODIFY’s are the problem? In
> EC> theory, it could also be the combination.
> EC>
>
> PH> sorry, I was wrong. Such a combination does exist in
> PH>the dpif_probe_feature, and it probes the datapath by trying to put
> flows:
> PH>
> PH> error = dpif_flow_put(dpif, DPIF_FP_CREATE | DPIF_FP_MODIFY |
> PH>DPIF_FP_PROBE,
> PH> key->data, key->size, NULL, 0,
> PH> nl_actions, nl_actions_size,
> PH> ufid, NON_PMD_CORE_ID, NULL);
>
> PH> so we CANNOT change the code into:
>
> PH> if (put->flags & DPIF_FP_CREATE && !(put->flags & DPIF_FP_MODIFY))
>
> PH> as the issue the patch tries to fix only exist in MODIFY alone path.
> PH> If CREATE bit is set, we need to go through the CREATE path no matter
> if
> PH> MODIFY exist or not.
>
> Ok, if this is only used by the probe function we should be fine. I did
> quickly search the code and it seems to be the way. However, if it’s ever
> used by any part of ovs with both flags set, it might fail the lookup and
> we run into the same problem.
>
By "the same problem", you mean the problem listed in this patch or this
fix might
[ovs-dev] [ovs-dev v12] ofproto-dpif-upcall: fix push_dp_ops
push_dp_ops only handles delete ops errors but ignores the modify
ops results. It's better to handle all the dp operation errors in
a consistent way.
This patch prevents the inconsistency by considering modify failure
in revalidators.
To note, we cannot perform two state transitions and change ukey_state
into UKEY_EVICTED directly here, because, if we do so, the
sweep will remove the ukey alone and leave dp flow alive. Later, the
dump will retrieve the dp flow and might even recover it. This will
contribute the stats of this dp flow twice.
v8->v9: add testsuite and delete INCONSISTENT ukey at revalidate_sweep.
v9->v10: change the commit message and refine the test case.
v10->v11: fix indentation and refine the test case.
v11->v12: fix the test suite.
Signed-off-by: Peng He
---
ofproto/ofproto-dpif-upcall.c | 50 +--
tests/dpif-netdev.at | 43 ++
2 files changed, 79 insertions(+), 14 deletions(-)
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index 04b583f81..cde03abc6 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -53,6 +53,7 @@
VLOG_DEFINE_THIS_MODULE(ofproto_dpif_upcall);
COVERAGE_DEFINE(dumped_duplicate_flow);
+COVERAGE_DEFINE(dumped_inconsistent_flow);
COVERAGE_DEFINE(dumped_new_flow);
COVERAGE_DEFINE(handler_duplicate_upcall);
COVERAGE_DEFINE(revalidate_missed_dp_flow);
@@ -258,6 +259,7 @@ enum ukey_state {
UKEY_CREATED = 0,
UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is queued. */
UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed. */
+UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is inconsistent. */
UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is queued. */
UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
UKEY_DELETED, /* Ukey removed from umap, ukey free is deferred. */
@@ -1999,6 +2001,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_VISIBLE -> UKEY_EVICTED
* A handler attempts to install the flow, but the datapath rejects it.
* Consider that the datapath has already destroyed it.
+ * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
+ * A revalidator modifies the flow with error returns.
+ * UKEY_INCONSISTENT -> UKEY_EVICTING
+ * A revalidator decides to evict the datapath flow.
* UKEY_OPERATIONAL -> UKEY_EVICTING
* A revalidator decides to evict the datapath flow.
* UKEY_EVICTING-> UKEY_EVICTED
@@ -2006,8 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_EVICTED -> UKEY_DELETED
* A revalidator has removed the ukey from the umap and is deleting it.
*/
-if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
- dst < UKEY_DELETED)) {
+if (ukey->state == dst - 1 ||
+ (ukey->state == UKEY_VISIBLE && dst < UKEY_DELETED) ||
+ (ukey->state == UKEY_OPERATIONAL && dst == UKEY_EVICTING)) {
ukey->state = dst;
} else {
struct ds ds = DS_EMPTY_INITIALIZER;
@@ -2490,26 +2497,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops,
size_t n_ops)
for (i = 0; i < n_ops; i++) {
struct ukey_op *op = [i];
-struct dpif_flow_stats *push, *stats, push_buf;
-
-stats = op->dop.flow_del.stats;
-push = _buf;
-
-if (op->dop.type != DPIF_OP_FLOW_DEL) {
-/* Only deleted flows need their stats pushed. */
-continue;
-}
if (op->dop.error) {
-/* flow_del error, 'stats' is unusable. */
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
-transition_ukey(op->ukey, UKEY_EVICTED);
+if (op->dop.type == DPIF_OP_FLOW_DEL) {
+transition_ukey(op->ukey, UKEY_EVICTED);
+} else {
+/* Modification of the flow failed. */
+transition_ukey(op->ukey, UKEY_INCONSISTENT);
+}
ovs_mutex_unlock(>ukey->mutex);
}
continue;
}
+if (op->dop.type != DPIF_OP_FLOW_DEL) {
+/* Only deleted flows need their stats pushed. */
+continue;
+}
+
+struct dpif_flow_stats *push, *stats, push_buf;
+
+stats = op->dop.flow_del.stats;
+push = _buf;
+
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
transition_ukey(op->ukey, UKEY_EVICTED);
@@ -2857,6 +2869,15 @@ revalidate(struct revalidator *revalidator)
continue;
}
+if (ukey->state == UKEY_INCONSISTENT) {
+ukey->dump_seq = dump_seq;
+
[ovs-dev] [ovs-dev v12] ofproto-dpif-upcall: fix push_dp_ops
push_dp_ops only handles delete ops errors but ignores the modify
ops results. It's better to handle all the dp operation errors in
a consistent way.
This patch prevents the inconsistency by considering modify failure
in revalidators.
To note, we cannot perform two state transitions and change ukey_state
into UKEY_EVICTED directly here, because, if we do so, the
sweep will remove the ukey alone and leave dp flow alive. Later, the
dump will retrieve the dp flow and might even recover it. This will
contribute the stats of this dp flow twice.
v8->v9: add testsuite and delete INCONSISTENT ukey at revalidate_sweep.
v9->v10: change the commit message and refine the test case.
v10->v11: fix indentation and refine the test case.
v11->v12: fix the test suite.
Signed-off-by: Peng He
---
ofproto/ofproto-dpif-upcall.c | 50 +--
tests/dpif-netdev.at | 45 +++
2 files changed, 81 insertions(+), 14 deletions(-)
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index 04b583f81..cde03abc6 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -53,6 +53,7 @@
VLOG_DEFINE_THIS_MODULE(ofproto_dpif_upcall);
COVERAGE_DEFINE(dumped_duplicate_flow);
+COVERAGE_DEFINE(dumped_inconsistent_flow);
COVERAGE_DEFINE(dumped_new_flow);
COVERAGE_DEFINE(handler_duplicate_upcall);
COVERAGE_DEFINE(revalidate_missed_dp_flow);
@@ -258,6 +259,7 @@ enum ukey_state {
UKEY_CREATED = 0,
UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is queued. */
UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed. */
+UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is inconsistent. */
UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is queued. */
UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
UKEY_DELETED, /* Ukey removed from umap, ukey free is deferred. */
@@ -1999,6 +2001,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_VISIBLE -> UKEY_EVICTED
* A handler attempts to install the flow, but the datapath rejects it.
* Consider that the datapath has already destroyed it.
+ * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
+ * A revalidator modifies the flow with error returns.
+ * UKEY_INCONSISTENT -> UKEY_EVICTING
+ * A revalidator decides to evict the datapath flow.
* UKEY_OPERATIONAL -> UKEY_EVICTING
* A revalidator decides to evict the datapath flow.
* UKEY_EVICTING-> UKEY_EVICTED
@@ -2006,8 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_EVICTED -> UKEY_DELETED
* A revalidator has removed the ukey from the umap and is deleting it.
*/
-if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
- dst < UKEY_DELETED)) {
+if (ukey->state == dst - 1 ||
+ (ukey->state == UKEY_VISIBLE && dst < UKEY_DELETED) ||
+ (ukey->state == UKEY_OPERATIONAL && dst == UKEY_EVICTING)) {
ukey->state = dst;
} else {
struct ds ds = DS_EMPTY_INITIALIZER;
@@ -2490,26 +2497,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops,
size_t n_ops)
for (i = 0; i < n_ops; i++) {
struct ukey_op *op = [i];
-struct dpif_flow_stats *push, *stats, push_buf;
-
-stats = op->dop.flow_del.stats;
-push = _buf;
-
-if (op->dop.type != DPIF_OP_FLOW_DEL) {
-/* Only deleted flows need their stats pushed. */
-continue;
-}
if (op->dop.error) {
-/* flow_del error, 'stats' is unusable. */
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
-transition_ukey(op->ukey, UKEY_EVICTED);
+if (op->dop.type == DPIF_OP_FLOW_DEL) {
+transition_ukey(op->ukey, UKEY_EVICTED);
+} else {
+/* Modification of the flow failed. */
+transition_ukey(op->ukey, UKEY_INCONSISTENT);
+}
ovs_mutex_unlock(>ukey->mutex);
}
continue;
}
+if (op->dop.type != DPIF_OP_FLOW_DEL) {
+/* Only deleted flows need their stats pushed. */
+continue;
+}
+
+struct dpif_flow_stats *push, *stats, push_buf;
+
+stats = op->dop.flow_del.stats;
+push = _buf;
+
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
transition_ukey(op->ukey, UKEY_EVICTED);
@@ -2857,6 +2869,15 @@ revalidate(struct revalidator *revalidator)
continue;
}
+if (ukey->state == UKEY_INCONSISTENT) {
+ukey->dump_seq = dump_seq;
+
[ovs-dev] [ovs-dev v12] ofproto-dpif-upcall: fix push_dp_ops
push_dp_ops only handles delete ops errors but ignores the modify
ops results. It's better to handle all the dp operation errors in
a consistent way.
This patch prevents the inconsistency by considering modify failure
in revalidators.
To note, we cannot perform two state transitions and change ukey_state
into UKEY_EVICTED directly here, because, if we do so, the
sweep will remove the ukey alone and leave dp flow alive. Later, the
dump will retrieve the dp flow and might even recover it. This will
contribute the stats of this dp flow twice.
v8->v9: add testsuite and delete INCONSISTENT ukey at revalidate_sweep.
v9->v10: change the commit message and refine the test case.
v10->v11: fix indentation and refine the test case.
v11->v12: fix the test suite.
Signed-off-by: Peng He
---
ofproto/ofproto-dpif-upcall.c | 50 +--
tests/dpif-netdev.at | 44 ++
2 files changed, 80 insertions(+), 14 deletions(-)
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index cd57fdbd9..c7df2d2f0 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -53,6 +53,7 @@
VLOG_DEFINE_THIS_MODULE(ofproto_dpif_upcall);
COVERAGE_DEFINE(dumped_duplicate_flow);
+COVERAGE_DEFINE(dumped_inconsistent_flow);
COVERAGE_DEFINE(dumped_new_flow);
COVERAGE_DEFINE(handler_duplicate_upcall);
COVERAGE_DEFINE(revalidate_missed_dp_flow);
@@ -258,6 +259,7 @@ enum ukey_state {
UKEY_CREATED = 0,
UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is queued. */
UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed. */
+UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is inconsistent. */
UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is queued. */
UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
UKEY_DELETED, /* Ukey removed from umap, ukey free is deferred. */
@@ -1999,6 +2001,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_VISIBLE -> UKEY_EVICTED
* A handler attempts to install the flow, but the datapath rejects it.
* Consider that the datapath has already destroyed it.
+ * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
+ * A revalidator modifies the flow with error returns.
+ * UKEY_INCONSISTENT -> UKEY_EVICTING
+ * A revalidator decides to evict the datapath flow.
* UKEY_OPERATIONAL -> UKEY_EVICTING
* A revalidator decides to evict the datapath flow.
* UKEY_EVICTING-> UKEY_EVICTED
@@ -2006,8 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_EVICTED -> UKEY_DELETED
* A revalidator has removed the ukey from the umap and is deleting it.
*/
-if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
- dst < UKEY_DELETED)) {
+if (ukey->state == dst - 1 ||
+ (ukey->state == UKEY_VISIBLE && dst < UKEY_DELETED) ||
+ (ukey->state == UKEY_OPERATIONAL && dst == UKEY_EVICTING)) {
ukey->state = dst;
} else {
struct ds ds = DS_EMPTY_INITIALIZER;
@@ -2472,26 +2479,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops,
size_t n_ops)
for (i = 0; i < n_ops; i++) {
struct ukey_op *op = [i];
-struct dpif_flow_stats *push, *stats, push_buf;
-
-stats = op->dop.flow_del.stats;
-push = _buf;
-
-if (op->dop.type != DPIF_OP_FLOW_DEL) {
-/* Only deleted flows need their stats pushed. */
-continue;
-}
if (op->dop.error) {
-/* flow_del error, 'stats' is unusable. */
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
-transition_ukey(op->ukey, UKEY_EVICTED);
+if (op->dop.type == DPIF_OP_FLOW_DEL) {
+transition_ukey(op->ukey, UKEY_EVICTED);
+} else {
+/* Modification of the flow failed. */
+transition_ukey(op->ukey, UKEY_INCONSISTENT);
+}
ovs_mutex_unlock(>ukey->mutex);
}
continue;
}
+if (op->dop.type != DPIF_OP_FLOW_DEL) {
+/* Only deleted flows need their stats pushed. */
+continue;
+}
+
+struct dpif_flow_stats *push, *stats, push_buf;
+
+stats = op->dop.flow_del.stats;
+push = _buf;
+
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
transition_ukey(op->ukey, UKEY_EVICTED);
@@ -2839,6 +2851,15 @@ revalidate(struct revalidator *revalidator)
continue;
}
+if (ukey->state == UKEY_INCONSISTENT) {
+ukey->dump_seq = dump_seq;
+
Re: [ovs-dev] [ovs-dev v12] ofproto-dpif-upcall: fix push_dp_ops
I miss one command in the test suite. will resend.
Peng He 于2023年7月1日周六 12:50写道：
> push_dp_ops only handles delete ops errors but ignores the modify
> ops results. It's better to handle all the dp operation errors in
> a consistent way.
>
> This patch prevents the inconsistency by considering modify failure
> in revalidators.
>
> To note, we cannot perform two state transitions and change ukey_state
> into UKEY_EVICTED directly here, because, if we do so, the
> sweep will remove the ukey alone and leave dp flow alive. Later, the
> dump will retrieve the dp flow and might even recover it. This will
> contribute the stats of this dp flow twice.
>
> v8->v9: add testsuite and delete INCONSISTENT ukey at revalidate_sweep.
> v9->v10: change the commit message and refine the test case.
> v10->v11: fix indentation and refine the test case.
> v11->v12: fix the test suite.
>
> Signed-off-by: Peng He
> ---
> ofproto/ofproto-dpif-upcall.c | 50 +--
> tests/dpif-netdev.at | 44 ++
> 2 files changed, 80 insertions(+), 14 deletions(-)
>
> diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
> index cd57fdbd9..c7df2d2f0 100644
> --- a/ofproto/ofproto-dpif-upcall.c
> +++ b/ofproto/ofproto-dpif-upcall.c
> @@ -53,6 +53,7 @@
> VLOG_DEFINE_THIS_MODULE(ofproto_dpif_upcall);
>
> COVERAGE_DEFINE(dumped_duplicate_flow);
> +COVERAGE_DEFINE(dumped_inconsistent_flow);
> COVERAGE_DEFINE(dumped_new_flow);
> COVERAGE_DEFINE(handler_duplicate_upcall);
> COVERAGE_DEFINE(revalidate_missed_dp_flow);
> @@ -258,6 +259,7 @@ enum ukey_state {
> UKEY_CREATED = 0,
> UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is
> queued. */
> UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed. */
> +UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is
> inconsistent. */
> UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is
> queued. */
> UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
> UKEY_DELETED, /* Ukey removed from umap, ukey free is deferred.
> */
> @@ -1999,6 +2001,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
> ukey_state dst,
> * UKEY_VISIBLE -> UKEY_EVICTED
> * A handler attempts to install the flow, but the datapath rejects
> it.
> * Consider that the datapath has already destroyed it.
> + * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
> + * A revalidator modifies the flow with error returns.
> + * UKEY_INCONSISTENT -> UKEY_EVICTING
> + * A revalidator decides to evict the datapath flow.
> * UKEY_OPERATIONAL -> UKEY_EVICTING
> * A revalidator decides to evict the datapath flow.
> * UKEY_EVICTING-> UKEY_EVICTED
> @@ -2006,8 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
> ukey_state dst,
> * UKEY_EVICTED -> UKEY_DELETED
> * A revalidator has removed the ukey from the umap and is deleting
> it.
> */
> -if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
> - dst < UKEY_DELETED)) {
> +if (ukey->state == dst - 1 ||
> + (ukey->state == UKEY_VISIBLE && dst < UKEY_DELETED) ||
> + (ukey->state == UKEY_OPERATIONAL && dst == UKEY_EVICTING)) {
> ukey->state = dst;
> } else {
> struct ds ds = DS_EMPTY_INITIALIZER;
> @@ -2472,26 +2479,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op
> *ops, size_t n_ops)
>
> for (i = 0; i < n_ops; i++) {
> struct ukey_op *op = [i];
> -struct dpif_flow_stats *push, *stats, push_buf;
> -
> -stats = op->dop.flow_del.stats;
> -push = _buf;
> -
> -if (op->dop.type != DPIF_OP_FLOW_DEL) {
> -/* Only deleted flows need their stats pushed. */
> -continue;
> -}
>
> if (op->dop.error) {
> -/* flow_del error, 'stats' is unusable. */
> if (op->ukey) {
> ovs_mutex_lock(>ukey->mutex);
> -transition_ukey(op->ukey, UKEY_EVICTED);
> +if (op->dop.type == DPIF_OP_FLOW_DEL) {
> +transition_ukey(op->ukey, UKEY_EVICTED);
> +} else {
> +/* Modification of the flow failed. */
> +transition_ukey(op->ukey, UKEY_INCONSISTENT);
> +}
> ovs_mutex_unlock(>ukey->mutex);
> }
> continue;
> }
>
> +if (op->dop.type != DP
[ovs-dev] [ovs-dev v12] ofproto-dpif-upcall: fix push_dp_ops
push_dp_ops only handles delete ops errors but ignores the modify
ops results. It's better to handle all the dp operation errors in
a consistent way.
This patch prevents the inconsistency by considering modify failure
in revalidators.
To note, we cannot perform two state transitions and change ukey_state
into UKEY_EVICTED directly here, because, if we do so, the
sweep will remove the ukey alone and leave dp flow alive. Later, the
dump will retrieve the dp flow and might even recover it. This will
contribute the stats of this dp flow twice.
v8->v9: add testsuite and delete INCONSISTENT ukey at revalidate_sweep.
v9->v10: change the commit message and refine the test case.
v10->v11: fix indentation and refine the test case.
v11->v12: fix the test suite.
Signed-off-by: Peng He
---
ofproto/ofproto-dpif-upcall.c | 50 +--
tests/dpif-netdev.at | 44 ++
2 files changed, 80 insertions(+), 14 deletions(-)
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index cd57fdbd9..c7df2d2f0 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -53,6 +53,7 @@
VLOG_DEFINE_THIS_MODULE(ofproto_dpif_upcall);
COVERAGE_DEFINE(dumped_duplicate_flow);
+COVERAGE_DEFINE(dumped_inconsistent_flow);
COVERAGE_DEFINE(dumped_new_flow);
COVERAGE_DEFINE(handler_duplicate_upcall);
COVERAGE_DEFINE(revalidate_missed_dp_flow);
@@ -258,6 +259,7 @@ enum ukey_state {
UKEY_CREATED = 0,
UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is queued. */
UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed. */
+UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is inconsistent. */
UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is queued. */
UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
UKEY_DELETED, /* Ukey removed from umap, ukey free is deferred. */
@@ -1999,6 +2001,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_VISIBLE -> UKEY_EVICTED
* A handler attempts to install the flow, but the datapath rejects it.
* Consider that the datapath has already destroyed it.
+ * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
+ * A revalidator modifies the flow with error returns.
+ * UKEY_INCONSISTENT -> UKEY_EVICTING
+ * A revalidator decides to evict the datapath flow.
* UKEY_OPERATIONAL -> UKEY_EVICTING
* A revalidator decides to evict the datapath flow.
* UKEY_EVICTING-> UKEY_EVICTED
@@ -2006,8 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_EVICTED -> UKEY_DELETED
* A revalidator has removed the ukey from the umap and is deleting it.
*/
-if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
- dst < UKEY_DELETED)) {
+if (ukey->state == dst - 1 ||
+ (ukey->state == UKEY_VISIBLE && dst < UKEY_DELETED) ||
+ (ukey->state == UKEY_OPERATIONAL && dst == UKEY_EVICTING)) {
ukey->state = dst;
} else {
struct ds ds = DS_EMPTY_INITIALIZER;
@@ -2472,26 +2479,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops,
size_t n_ops)
for (i = 0; i < n_ops; i++) {
struct ukey_op *op = [i];
-struct dpif_flow_stats *push, *stats, push_buf;
-
-stats = op->dop.flow_del.stats;
-push = _buf;
-
-if (op->dop.type != DPIF_OP_FLOW_DEL) {
-/* Only deleted flows need their stats pushed. */
-continue;
-}
if (op->dop.error) {
-/* flow_del error, 'stats' is unusable. */
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
-transition_ukey(op->ukey, UKEY_EVICTED);
+if (op->dop.type == DPIF_OP_FLOW_DEL) {
+transition_ukey(op->ukey, UKEY_EVICTED);
+} else {
+/* Modification of the flow failed. */
+transition_ukey(op->ukey, UKEY_INCONSISTENT);
+}
ovs_mutex_unlock(>ukey->mutex);
}
continue;
}
+if (op->dop.type != DPIF_OP_FLOW_DEL) {
+/* Only deleted flows need their stats pushed. */
+continue;
+}
+
+struct dpif_flow_stats *push, *stats, push_buf;
+
+stats = op->dop.flow_del.stats;
+push = _buf;
+
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
transition_ukey(op->ukey, UKEY_EVICTED);
@@ -2839,6 +2851,15 @@ revalidate(struct revalidator *revalidator)
continue;
}
+if (ukey->state == UKEY_INCONSISTENT) {
+ukey->dump_seq = dump_seq;
+
[ovs-dev] [ovs-dev v4] dpif-netdev: fix dpif_netdev_flow_put
OVS allows overlapping megaflows, as long as the actions of these
megaflows are equal. However, the current implementation of action
modification relies on flow_lookup instead of ufid, this could result
in looking up a wrong megaflow and make the ukeys and megaflows inconsistent
Just like the test case in the patch, at first we have a rule with the
prefix:
10.1.2.0/24
and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with IP
10.1.2.2 is received.
Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep the
10.1.2.2/24 megaflow and just changes its action instead of extending
the prefix into 10.1.2.2/16.
then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
this time, we will have an overlapping megaflow with the right prefix:
10.1.0.2/16
now we have two megaflows:
10.1.2.2/24
10.1.0.2/16
last, suppose we have changed the ruleset again. The revalidator this
time still decides to change the actions of both megaflows instead of
deleting them.
The dpif_netdev_flow_put will search the megaflow to modify with unmasked
keys, however it might lookup the wrong megaflow as the key 10.1.2.2 matches
both 10.1.2.2/24 and 10.1.0.2/16!
This patch changes the megaflow lookup code in modification path into
relying the ufid to find the correct megaflow instead of key lookup.
Signed-off-by: Peng He
---
lib/dpif-netdev.c | 62 ++-
tests/pmd.at | 48
2 files changed, 82 insertions(+), 28 deletions(-)
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index 70b953ae6..b90ed1542 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -4198,36 +4198,43 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
memset(stats, 0, sizeof *stats);
}
-ovs_mutex_lock(>flow_mutex);
-netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
-if (!netdev_flow) {
-if (put->flags & DPIF_FP_CREATE) {
+if (put->flags & DPIF_FP_CREATE) {
+ovs_mutex_lock(>flow_mutex);
+netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
+if (!netdev_flow) {
dp_netdev_flow_add(pmd, match, ufid, put->actions,
put->actions_len, ODPP_NONE);
} else {
-error = ENOENT;
+error = EEXIST;
}
+ovs_mutex_unlock(>flow_mutex);
} else {
+netdev_flow = dp_netdev_pmd_find_flow(pmd, ufid,
+ put->key, put->key_len);
+
if (put->flags & DPIF_FP_MODIFY) {
-struct dp_netdev_actions *new_actions;
-struct dp_netdev_actions *old_actions;
+if (!netdev_flow) {
+error = ENOENT;
+} else {
+struct dp_netdev_actions *new_actions;
+struct dp_netdev_actions *old_actions;
-new_actions = dp_netdev_actions_create(put->actions,
- put->actions_len);
+new_actions = dp_netdev_actions_create(put->actions,
+ put->actions_len);
-old_actions = dp_netdev_flow_get_actions(netdev_flow);
-ovsrcu_set(_flow->actions, new_actions);
+old_actions = dp_netdev_flow_get_actions(netdev_flow);
+ovsrcu_set(_flow->actions, new_actions);
-queue_netdev_flow_put(pmd, netdev_flow, match,
- put->actions, put->actions_len,
- DP_NETDEV_FLOW_OFFLOAD_OP_MOD);
-log_netdev_flow_change(netdev_flow, match, old_actions,
- put->actions, put->actions_len);
+queue_netdev_flow_put(pmd, netdev_flow, match,
+ put->actions, put->actions_len,
+ DP_NETDEV_FLOW_OFFLOAD_OP_MOD);
+log_netdev_flow_change(netdev_flow, match, old_actions,
+ put->actions, put->actions_len);
-if (stats) {
-get_dpif_flow_status(pmd->dp, netdev_flow, stats, NULL);
-}
-if (put->flags & DPIF_FP_ZERO_STATS) {
+if (stats) {
+get_dpif_flow_status(pmd->dp, netdev_flow, stats, NULL);
+}
+if (put->flags & DPIF_FP_ZERO_STATS) {
/* XXX: The userspace datapath uses thread local statistics
* (for flows), which should be updated only by the owning
* thread. Since we cannot write on stats memory here,
@@ -4237,18 +4244,17 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
* - Should the need arise, this operation can be implemented
* by keeping a base value (t
Re: [ovs-dev] [ovs-dev v3] dpif-netdev: fix dpif_netdev_flow_put
Eelco Chaudron 于2023年6月23日周五 22:31写道：
>
>
> On 15 Jun 2023, at 4:51, Peng He wrote:
>
> > OVS allows overlapping megaflows, as long as the actions of these
> > megaflows are equal. However, the current implementation of action
> > modification relies on flow_lookup instead of ufid, this could result
> > in looking up a wrong megaflow and make the ukeys and megaflows
> inconsistent
> >
> > Just like the test case in the patch, at first we have a rule with the
> > prefix:
> >
> > 10.1.2.0/24
> >
> > and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with
> IP
> > 10.1.2.2 is received.
> >
> > Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep
> the
> > 10.1.2.2/24 megaflow and just changes its action instead of extending
> > the prefix into 10.1.2.2/16.
> >
> > then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
> > this time, we will have an overlapping megaflow with the right prefix:
> > 10.1.0.2/16
> >
> > now we have two megaflows:
> > 10.1.2.2/24
> > 10.1.0.2/16
> >
> > last, suppose we have changed the ruleset again. The revalidator this
> > time still decides to change the actions of both megaflows instead of
> > deleting them.
> >
> > The dpif_netdev_flow_put will search the megaflow to modify with unmasked
> > keys, however it might lookup the wrong megaflow as the key 10.1.2.2
> matches
> > both 10.1.2.2/24 and 10.1.0.2/16!
> >
> > This patch changes the megaflow lookup code in modification path into
> > relying the ufid to find the correct megaflow instead of key lookup.
> >
> > Signed-off-by: Peng He
>
> Thanks for the fix, and in general it looks good, however I have one
> question about not covering all cases. See below.
>
> Cheers,
>
> Eelco
>
>
> > ---
> > lib/dpif-netdev.c | 62 ++-
> > tests/pmd.at | 48
> > 2 files changed, 82 insertions(+), 28 deletions(-)
> >
> > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> > index 70b953ae6..b90ed1542 100644
> > --- a/lib/dpif-netdev.c
> > +++ b/lib/dpif-netdev.c
> > @@ -4198,36 +4198,43 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
> > memset(stats, 0, sizeof *stats);
> > }
> >
> > -ovs_mutex_lock(>flow_mutex);
> > -netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > -if (!netdev_flow) {
> > -if (put->flags & DPIF_FP_CREATE) {
> > +if (put->flags & DPIF_FP_CREATE) {
>
> Should this not be:
>
> if (put->flags & DPIF_FP_CREATE && !(put->flags & DPIF_FP_MODIFY))
>
> I know this will break the fix, but I’m wondering what the possible
> combinations are.
> Quickly looking at the code the following ones seem to exist:
>
> DPIF_FP_CREATE -> Create a flow, if one exists return EEXIST
> DPIF_FP_MODIFY -> Modify a flow, if it does not exist return ENONT
> DPIF_FP_CREATE | DPIF_FP_MODIFY -> If it exists modify it, if it does
> not exists create it.
>
> Could the last combination not potentially fail the lookup?
>
> Or are we assuming only standalone DPIF_FP_MODIFY’s are the problem? In
> theory, it could also be the combination.
>
sorry, I was wrong. Such a combination does exist in
the dpif_probe_feature, and it probes the datapath by trying to put flows:
error = dpif_flow_put(dpif, DPIF_FP_CREATE | DPIF_FP_MODIFY |
DPIF_FP_PROBE,
key->data, key->size, NULL, 0,
nl_actions, nl_actions_size,
ufid, NON_PMD_CORE_ID, NULL);
so we CANNOT change the code into:
if (put->flags & DPIF_FP_CREATE && !(put->flags & DPIF_FP_MODIFY))
as the issue the patch tries to fix only exist in MODIFY alone path.
If CREATE bit is set, we need to go through the CREATE path no matter if
MODIFY exist or not.
>
> > +ovs_mutex_lock(>flow_mutex);
> > +netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > +if (!netdev_flow) {
> > dp_netdev_flow_add(pmd, match, ufid, put->actions,
> > put->actions_len, ODPP_NONE);
> > } else {
> > -error = ENOENT;
> > +error = EEXIST;
> > }
> > +ovs_mutex_unlock(>flow_mutex);
> > } else {
> > +netdev_flow = dp_netdev_pmd_find_flow(pmd, ufid,
> > + put->key, put->key_len);
> >
Re: [ovs-dev] [ovs-dev v11] ofproto-dpif-upcall: fix push_dp_ops
Ilya Maximets 于2023年6月23日周五 20:40写道：
> On 6/23/23 14:20, Eelco Chaudron wrote:
> >
> >
> > On 9 Jun 2023, at 17:03, Peng He wrote:
> >
> >> push_dp_ops only handles delete ops errors but ignores the modify
> >> ops results. It's better to handle all the dp operation errors in
> >> a consistent way.
> >>
> >> This patch prevents the inconsistency by considering modify failure
> >> in revalidators.
> >>
> >> To note, we cannot perform two state transitions and change ukey_state
> >> into UKEY_EVICTED directly here, because, if we do so, the
> >> sweep will remove the ukey alone and leave dp flow alive. Later, the
> >> dump will retrieve the dp flow and might even recover it. This will
> >> contribute the stats of this dp flow twice.
> >>
> >> v8->v9: add testsuite and delete INCONSISTENT ukey at
> revalidate_sweep.
> >> v9->v10: change the commit message and refine the test case.
> >> v10->v11: fix indentation and refine the test case.
> >>
> >> Signed-off-by: Peng He
> >
> > Thanks for making these changes, there is a merge conflict on the tests
> case and some small comments below.
> >
> > So maybe you can send a v12 and I’ll do a quick check and ACK it. What
> do you think?
> >
> > Cheers,
> >
> > Eelco
> >
> >
> >> ---
> >> ofproto/ofproto-dpif-upcall.c | 51 +--
> >> tests/dpif-netdev.at | 44 ++
> >> 2 files changed, 81 insertions(+), 14 deletions(-)
> >>
> >> diff --git a/ofproto/ofproto-dpif-upcall.c
> b/ofproto/ofproto-dpif-upcall.c
> >> index cd57fdbd9..c920c749c 100644
> >> --- a/ofproto/ofproto-dpif-upcall.c
> >> +++ b/ofproto/ofproto-dpif-upcall.c
> >> @@ -62,6 +62,7 @@ COVERAGE_DEFINE(upcall_flow_limit_hit);
> >> COVERAGE_DEFINE(upcall_flow_limit_kill);
> >> COVERAGE_DEFINE(upcall_ukey_contention);
> >> COVERAGE_DEFINE(upcall_ukey_replace);
> >> +COVERAGE_DEFINE(dumped_inconsistent_flow);
> >
> > Can you add this in alphabetical order?
> >
> >>
> >> /* A thread that reads upcalls from dpif, forwards each upcall's
> packet,
> >> * and possibly sets up a kernel flow as a cache. */
> >> @@ -258,6 +259,7 @@ enum ukey_state {
> >> UKEY_CREATED = 0,
> >> UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is
> queued. */
> >> UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is
> installed. */
> >> +UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is
> inconsistent. */
> >> UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is
> queued. */
> >> UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted.
> */
> >> UKEY_DELETED, /* Ukey removed from umap, ukey free is
> deferred. */
> >> @@ -1999,6 +2001,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
> ukey_state dst,
> >> * UKEY_VISIBLE -> UKEY_EVICTED
> >> * A handler attempts to install the flow, but the datapath
> rejects it.
> >> * Consider that the datapath has already destroyed it.
> >> + * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
> >> + * A revalidator modifies the flow with error returns.
> >> + * UKEY_INCONSISTENT -> UKEY_EVICTING
> >> + * A revalidator decides to evict the datapath flow.
> >> * UKEY_OPERATIONAL -> UKEY_EVICTING
> >> * A revalidator decides to evict the datapath flow.
> >> * UKEY_EVICTING-> UKEY_EVICTED
> >> @@ -2006,8 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
> ukey_state dst,
> >> * UKEY_EVICTED -> UKEY_DELETED
> >> * A revalidator has removed the ukey from the umap and is
> deleting it.
> >> */
> >> -if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
> >> - dst < UKEY_DELETED)) {
> >> +if (ukey->state == dst - 1 ||
> >> + (ukey->state == UKEY_VISIBLE && dst < UKEY_DELETED) ||
> >> + (ukey->state == UKEY_OPERATIONAL && dst == UKEY_EVICTING)) {
> >> ukey->state = dst;
> >> } else {
> >> struct ds ds = DS_EMPTY_INITIALIZER;
> >> @@ -2472,26 +2479,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op
> *ops, size_t n_ops)
> >>
> >> for (i
Re: [ovs-dev] [ovs-dev v3] dpif-netdev: fix dpif_netdev_flow_put
Ilya Maximets 于2023年6月24日周六 04:26写道：
> On 6/15/23 04:51, Peng He wrote:
> > OVS allows overlapping megaflows, as long as the actions of these
> > megaflows are equal. However, the current implementation of action
> > modification relies on flow_lookup instead of ufid, this could result
> > in looking up a wrong megaflow and make the ukeys and megaflows
> inconsistent
> >
> > Just like the test case in the patch, at first we have a rule with the
> > prefix:
> >
> > 10.1.2.0/24
> >
> > and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with
> IP
> > 10.1.2.2 is received.
> >
> > Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep
> the
> > 10.1.2.2/24 megaflow and just changes its action instead of extending
> > the prefix into 10.1.2.2/16.
> >
> > then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
> > this time, we will have an overlapping megaflow with the right prefix:
> > 10.1.0.2/16
> >
> > now we have two megaflows:
> > 10.1.2.2/24
> > 10.1.0.2/16
> >
> > last, suppose we have changed the ruleset again. The revalidator this
> > time still decides to change the actions of both megaflows instead of
> > deleting them.
> >
> > The dpif_netdev_flow_put will search the megaflow to modify with unmasked
> > keys, however it might lookup the wrong megaflow as the key 10.1.2.2
> matches
> > both 10.1.2.2/24 and 10.1.0.2/16!
> >
> > This patch changes the megaflow lookup code in modification path into
> > relying the ufid to find the correct megaflow instead of key lookup.
> >
> > Signed-off-by: Peng He
> > ---
> > lib/dpif-netdev.c | 62 ++-
> > tests/pmd.at | 48
> > 2 files changed, 82 insertions(+), 28 deletions(-)
> >
> > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> > index 70b953ae6..b90ed1542 100644
> > --- a/lib/dpif-netdev.c
> > +++ b/lib/dpif-netdev.c
> > @@ -4198,36 +4198,43 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
> > memset(stats, 0, sizeof *stats);
> > }
> >
> > -ovs_mutex_lock(>flow_mutex);
> > -netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > -if (!netdev_flow) {
> > -if (put->flags & DPIF_FP_CREATE) {
> > +if (put->flags & DPIF_FP_CREATE) {
> > +ovs_mutex_lock(>flow_mutex);
> > +netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > +if (!netdev_flow) {
> > dp_netdev_flow_add(pmd, match, ufid, put->actions,
> > put->actions_len, ODPP_NONE);
> > } else {
> > -error = ENOENT;
> > +error = EEXIST;
> > }
> > +ovs_mutex_unlock(>flow_mutex);
> > } else {
> > +netdev_flow = dp_netdev_pmd_find_flow(pmd, ufid,
> > + put->key, put->key_len);
> > +
> > if (put->flags & DPIF_FP_MODIFY) {
> > -struct dp_netdev_actions *new_actions;
> > -struct dp_netdev_actions *old_actions;
> > +if (!netdev_flow) {
> > +error = ENOENT;
> > +} else {
> > +struct dp_netdev_actions *new_actions;
> > +struct dp_netdev_actions *old_actions;
> >
> > -new_actions = dp_netdev_actions_create(put->actions,
> > - put->actions_len);
> > +new_actions = dp_netdev_actions_create(put->actions,
> > +
> put->actions_len);
> >
> > -old_actions = dp_netdev_flow_get_actions(netdev_flow);
> > -ovsrcu_set(_flow->actions, new_actions);
> > +old_actions = dp_netdev_flow_get_actions(netdev_flow);
> > +ovsrcu_set(_flow->actions, new_actions);
> >
> > -queue_netdev_flow_put(pmd, netdev_flow, match,
> > - put->actions, put->actions_len,
> > - DP_NETDEV_FLOW_OFFLOAD_OP_MOD);
> > -log_netdev_flow_change(netdev_flow, match, old_actions,
> > - put->actions, put->actions_len);
> > +queue_netdev_flow_put(pmd, netdev_flow, match,
> > + put->actions, put->actions_len,
> > +
[ovs-dev] [ovs-dev v3] dpif-netdev: fix dpif_netdev_flow_put
OVS allows overlapping megaflows, as long as the actions of these
megaflows are equal. However, the current implementation of action
modification relies on flow_lookup instead of ufid, this could result
in looking up a wrong megaflow and make the ukeys and megaflows inconsistent
Just like the test case in the patch, at first we have a rule with the
prefix:
10.1.2.0/24
and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with IP
10.1.2.2 is received.
Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep the
10.1.2.2/24 megaflow and just changes its action instead of extending
the prefix into 10.1.2.2/16.
then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
this time, we will have an overlapping megaflow with the right prefix:
10.1.0.2/16
now we have two megaflows:
10.1.2.2/24
10.1.0.2/16
last, suppose we have changed the ruleset again. The revalidator this
time still decides to change the actions of both megaflows instead of
deleting them.
The dpif_netdev_flow_put will search the megaflow to modify with unmasked
keys, however it might lookup the wrong megaflow as the key 10.1.2.2 matches
both 10.1.2.2/24 and 10.1.0.2/16!
This patch changes the megaflow lookup code in modification path into
relying the ufid to find the correct megaflow instead of key lookup.
Signed-off-by: Peng He
---
lib/dpif-netdev.c | 62 ++-
tests/pmd.at | 48
2 files changed, 82 insertions(+), 28 deletions(-)
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index 70b953ae6..b90ed1542 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -4198,36 +4198,43 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
memset(stats, 0, sizeof *stats);
}
-ovs_mutex_lock(>flow_mutex);
-netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
-if (!netdev_flow) {
-if (put->flags & DPIF_FP_CREATE) {
+if (put->flags & DPIF_FP_CREATE) {
+ovs_mutex_lock(>flow_mutex);
+netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
+if (!netdev_flow) {
dp_netdev_flow_add(pmd, match, ufid, put->actions,
put->actions_len, ODPP_NONE);
} else {
-error = ENOENT;
+error = EEXIST;
}
+ovs_mutex_unlock(>flow_mutex);
} else {
+netdev_flow = dp_netdev_pmd_find_flow(pmd, ufid,
+ put->key, put->key_len);
+
if (put->flags & DPIF_FP_MODIFY) {
-struct dp_netdev_actions *new_actions;
-struct dp_netdev_actions *old_actions;
+if (!netdev_flow) {
+error = ENOENT;
+} else {
+struct dp_netdev_actions *new_actions;
+struct dp_netdev_actions *old_actions;
-new_actions = dp_netdev_actions_create(put->actions,
- put->actions_len);
+new_actions = dp_netdev_actions_create(put->actions,
+ put->actions_len);
-old_actions = dp_netdev_flow_get_actions(netdev_flow);
-ovsrcu_set(_flow->actions, new_actions);
+old_actions = dp_netdev_flow_get_actions(netdev_flow);
+ovsrcu_set(_flow->actions, new_actions);
-queue_netdev_flow_put(pmd, netdev_flow, match,
- put->actions, put->actions_len,
- DP_NETDEV_FLOW_OFFLOAD_OP_MOD);
-log_netdev_flow_change(netdev_flow, match, old_actions,
- put->actions, put->actions_len);
+queue_netdev_flow_put(pmd, netdev_flow, match,
+ put->actions, put->actions_len,
+ DP_NETDEV_FLOW_OFFLOAD_OP_MOD);
+log_netdev_flow_change(netdev_flow, match, old_actions,
+ put->actions, put->actions_len);
-if (stats) {
-get_dpif_flow_status(pmd->dp, netdev_flow, stats, NULL);
-}
-if (put->flags & DPIF_FP_ZERO_STATS) {
+if (stats) {
+get_dpif_flow_status(pmd->dp, netdev_flow, stats, NULL);
+}
+if (put->flags & DPIF_FP_ZERO_STATS) {
/* XXX: The userspace datapath uses thread local statistics
* (for flows), which should be updated only by the owning
* thread. Since we cannot write on stats memory here,
@@ -4237,18 +4244,17 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
* - Should the need arise, this operation can be implemented
* by keeping a base value (t
Re: [ovs-dev] [ovs-dev v2] dpif-netdev: fix dpif_netdev_flow_put
Simon Horman 于2023年6月14日周三 20:37写道： > On Tue, Jun 13, 2023 at 09:41:42AM +0800, Peng He wrote: > > Hi, Simon, > > > > I guess it's not related, as my patch fixes the code in dpif-netdev.c it > > should has any impact on the ovsdb side. > > But I am not sure that. > > Is this test case a spike test case? Is there any testsuite log? > > * Please don't top post > > * I am unsure what a spike test is > > * The link I provided leads to logs > > [1] > https://github.com/ovsrobot/ovs/actions/runs/5238809864/jobs/9458063244#step:11:5573 > [2] > https://github.com/ovsrobot/ovs/suites/13526199944/artifacts/743672747 > > * It seems that my report was inaccurate, sorry about that. > > As pointed out by Ilya, > it was not the "compacting online - cluster that failed", > but rather the test that is added by your patch > > [3] https://mail.openvswitch.org/pipermail/ovs-dev/2023-June/405546.html Thanks for the information! I check out the log, it seems one megaflow has been missing. The reason might be one megaflow has been aged and removed. I will send a v3 to improve that. -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v2] dpif-netdev: fix dpif_netdev_flow_put
Hi, Simon, I guess it's not related, as my patch fixes the code in dpif-netdev.c it should has any impact on the ovsdb side. But I am not sure that. Is this test case a spike test case? Is there any testsuite log? Simon Horman 于2023年6月12日周一 22:47写道： > On Mon, Jun 12, 2023 at 02:10:17AM +0000, Peng He wrote: > > OVS allows overlapping megaflows, as long as the actions of these > > megaflows are equal. However, the current implementation of action > > modification relies on flow_lookup instead of ufid, this could result > > in looking up a wrong megaflow and make the ukeys and megaflows > inconsistent > > > > Just like the test case in the patch, at first we have a rule with the > > prefix: > > > > 10.1.2.0/24 > > > > and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with > IP > > 10.1.2.2 is received. > > > > Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep > the > > 10.1.2.2/24 megaflow and just changes its action instead of extending > > the prefix into 10.1.2.2/16. > > > > then suppose we have a 10.1.0.2 packet, since it misses the megaflow, > > this time, we will have an overlapping megaflow with the right prefix: > > 10.1.0.2/16 > > > > now we have two megaflows: > > 10.1.2.2/24 > > 10.1.0.2/16 > > > > last, suppose we have changed the ruleset again. The revalidator this > > time still decides to change the actions of both megaflows instead of > > deleting them. > > > > The dpif_netdev_flow_put will search the megaflow to modify with unmasked > > keys, however it might lookup the wrong megaflow as the key 10.1.2.2 > matches > > both 10.1.2.2/24 and 10.1.0.2/16! > > > > This patch changes the megaflow lookup code in modification path into > > relying the ufid to find the correct megaflow instead of key lookup. > > > > Signed-off-by: Peng He > > Hi Peng, > > I see one failure in CI. It is the " compacting online - cluster " > test for "liunx clang test asan". Is this something we should be concerned > about? > > Link: > https://github.com/ovsrobot/ovs/actions/runs/5238809864/jobs/9458063244#step:11:5573 > -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v2] dpif-netdev: fix dpif_netdev_flow_put
OVS allows overlapping megaflows, as long as the actions of these
megaflows are equal. However, the current implementation of action
modification relies on flow_lookup instead of ufid, this could result
in looking up a wrong megaflow and make the ukeys and megaflows inconsistent
Just like the test case in the patch, at first we have a rule with the
prefix:
10.1.2.0/24
and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with IP
10.1.2.2 is received.
Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep the
10.1.2.2/24 megaflow and just changes its action instead of extending
the prefix into 10.1.2.2/16.
then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
this time, we will have an overlapping megaflow with the right prefix:
10.1.0.2/16
now we have two megaflows:
10.1.2.2/24
10.1.0.2/16
last, suppose we have changed the ruleset again. The revalidator this
time still decides to change the actions of both megaflows instead of
deleting them.
The dpif_netdev_flow_put will search the megaflow to modify with unmasked
keys, however it might lookup the wrong megaflow as the key 10.1.2.2 matches
both 10.1.2.2/24 and 10.1.0.2/16!
This patch changes the megaflow lookup code in modification path into
relying the ufid to find the correct megaflow instead of key lookup.
Signed-off-by: Peng He
---
lib/dpif-netdev.c | 62 ++-
tests/pmd.at | 46 +++
2 files changed, 80 insertions(+), 28 deletions(-)
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index 70b953ae6..b90ed1542 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -4198,36 +4198,43 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
memset(stats, 0, sizeof *stats);
}
-ovs_mutex_lock(>flow_mutex);
-netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
-if (!netdev_flow) {
-if (put->flags & DPIF_FP_CREATE) {
+if (put->flags & DPIF_FP_CREATE) {
+ovs_mutex_lock(>flow_mutex);
+netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
+if (!netdev_flow) {
dp_netdev_flow_add(pmd, match, ufid, put->actions,
put->actions_len, ODPP_NONE);
} else {
-error = ENOENT;
+error = EEXIST;
}
+ovs_mutex_unlock(>flow_mutex);
} else {
+netdev_flow = dp_netdev_pmd_find_flow(pmd, ufid,
+ put->key, put->key_len);
+
if (put->flags & DPIF_FP_MODIFY) {
-struct dp_netdev_actions *new_actions;
-struct dp_netdev_actions *old_actions;
+if (!netdev_flow) {
+error = ENOENT;
+} else {
+struct dp_netdev_actions *new_actions;
+struct dp_netdev_actions *old_actions;
-new_actions = dp_netdev_actions_create(put->actions,
- put->actions_len);
+new_actions = dp_netdev_actions_create(put->actions,
+ put->actions_len);
-old_actions = dp_netdev_flow_get_actions(netdev_flow);
-ovsrcu_set(_flow->actions, new_actions);
+old_actions = dp_netdev_flow_get_actions(netdev_flow);
+ovsrcu_set(_flow->actions, new_actions);
-queue_netdev_flow_put(pmd, netdev_flow, match,
- put->actions, put->actions_len,
- DP_NETDEV_FLOW_OFFLOAD_OP_MOD);
-log_netdev_flow_change(netdev_flow, match, old_actions,
- put->actions, put->actions_len);
+queue_netdev_flow_put(pmd, netdev_flow, match,
+ put->actions, put->actions_len,
+ DP_NETDEV_FLOW_OFFLOAD_OP_MOD);
+log_netdev_flow_change(netdev_flow, match, old_actions,
+ put->actions, put->actions_len);
-if (stats) {
-get_dpif_flow_status(pmd->dp, netdev_flow, stats, NULL);
-}
-if (put->flags & DPIF_FP_ZERO_STATS) {
+if (stats) {
+get_dpif_flow_status(pmd->dp, netdev_flow, stats, NULL);
+}
+if (put->flags & DPIF_FP_ZERO_STATS) {
/* XXX: The userspace datapath uses thread local statistics
* (for flows), which should be updated only by the owning
* thread. Since we cannot write on stats memory here,
@@ -4237,18 +4244,17 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
* - Should the need arise, this operation can be implemented
* by keeping a base value (t
[ovs-dev] [ovs-dev v11] ofproto-dpif-upcall: fix push_dp_ops
push_dp_ops only handles delete ops errors but ignores the modify
ops results. It's better to handle all the dp operation errors in
a consistent way.
This patch prevents the inconsistency by considering modify failure
in revalidators.
To note, we cannot perform two state transitions and change ukey_state
into UKEY_EVICTED directly here, because, if we do so, the
sweep will remove the ukey alone and leave dp flow alive. Later, the
dump will retrieve the dp flow and might even recover it. This will
contribute the stats of this dp flow twice.
v8->v9: add testsuite and delete INCONSISTENT ukey at revalidate_sweep.
v9->v10: change the commit message and refine the test case.
v10->v11: fix indentation and refine the test case.
Signed-off-by: Peng He
---
ofproto/ofproto-dpif-upcall.c | 51 +--
tests/dpif-netdev.at | 44 ++
2 files changed, 81 insertions(+), 14 deletions(-)
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index cd57fdbd9..c920c749c 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -62,6 +62,7 @@ COVERAGE_DEFINE(upcall_flow_limit_hit);
COVERAGE_DEFINE(upcall_flow_limit_kill);
COVERAGE_DEFINE(upcall_ukey_contention);
COVERAGE_DEFINE(upcall_ukey_replace);
+COVERAGE_DEFINE(dumped_inconsistent_flow);
/* A thread that reads upcalls from dpif, forwards each upcall's packet,
* and possibly sets up a kernel flow as a cache. */
@@ -258,6 +259,7 @@ enum ukey_state {
UKEY_CREATED = 0,
UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is queued. */
UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed. */
+UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is inconsistent. */
UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is queued. */
UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
UKEY_DELETED, /* Ukey removed from umap, ukey free is deferred. */
@@ -1999,6 +2001,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_VISIBLE -> UKEY_EVICTED
* A handler attempts to install the flow, but the datapath rejects it.
* Consider that the datapath has already destroyed it.
+ * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
+ * A revalidator modifies the flow with error returns.
+ * UKEY_INCONSISTENT -> UKEY_EVICTING
+ * A revalidator decides to evict the datapath flow.
* UKEY_OPERATIONAL -> UKEY_EVICTING
* A revalidator decides to evict the datapath flow.
* UKEY_EVICTING-> UKEY_EVICTED
@@ -2006,8 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_EVICTED -> UKEY_DELETED
* A revalidator has removed the ukey from the umap and is deleting it.
*/
-if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
- dst < UKEY_DELETED)) {
+if (ukey->state == dst - 1 ||
+ (ukey->state == UKEY_VISIBLE && dst < UKEY_DELETED) ||
+ (ukey->state == UKEY_OPERATIONAL && dst == UKEY_EVICTING)) {
ukey->state = dst;
} else {
struct ds ds = DS_EMPTY_INITIALIZER;
@@ -2472,26 +2479,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops,
size_t n_ops)
for (i = 0; i < n_ops; i++) {
struct ukey_op *op = [i];
-struct dpif_flow_stats *push, *stats, push_buf;
-
-stats = op->dop.flow_del.stats;
-push = _buf;
-
-if (op->dop.type != DPIF_OP_FLOW_DEL) {
-/* Only deleted flows need their stats pushed. */
-continue;
-}
if (op->dop.error) {
-/* flow_del error, 'stats' is unusable. */
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
-transition_ukey(op->ukey, UKEY_EVICTED);
+if (op->dop.type == DPIF_OP_FLOW_DEL) {
+transition_ukey(op->ukey, UKEY_EVICTED);
+} else {
+/* Modification of the flow failed */
+transition_ukey(op->ukey, UKEY_INCONSISTENT);
+}
ovs_mutex_unlock(>ukey->mutex);
}
continue;
}
+if (op->dop.type != DPIF_OP_FLOW_DEL) {
+/* Only deleted flows need their stats pushed. */
+continue;
+}
+
+struct dpif_flow_stats *push, *stats, push_buf;
+
+stats = op->dop.flow_del.stats;
+push = _buf;
+
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
transition_ukey(op->ukey, UKEY_EVICTED);
@@ -2839,6 +2851,16 @@ revalidate(struct revalidator *revalidator)
continue;
}
+if (ukey->state == UKEY_INCONSISTENT) {
+uk
Re: [ovs-dev] [ovs-dev v10] ofproto-dpif-upcall: fix push_dp_ops
Thanks for the reviews!
Eelco Chaudron 于2023年6月9日周五 20:47写道：
>
>
> On 3 Jun 2023, at 2:01, Peng He wrote:
>
> > push_dp_ops only handles delete ops errors but ignores the modify
> > ops results. It's better to handle all the dp operation errors in
> > a consistent way.
> >
> > This patch prevents the inconsistency by considering modify failure
> > in revalidators.
> >
> > To note, we cannot perform two state transitions and change ukey_state
> > into UKEY_EVICTED directly here, because, if we do so, the
> > sweep will remove the ukey alone and leave dp flow alive. Later, the
> > dump will retrieve the dp flow and might even recover it. This will
> > contribute the stats of this dp flow twice.
> >
> > v8->v9: add testsuite and delete INCONSISTENT ukey at revalidate_sweep.
> > v9->v10: change the commit message and refine the test case.
>
> Hi Peng,
>
> Thanks for keeping this patch updated :) Some more comments below, but I
> think the next rev should be all fine.
>
> Cheers,
>
> Eelco
>
> > Signed-off-by: Peng He
> > ---
> > ofproto/ofproto-dpif-upcall.c | 48 --
> > tests/dpif-netdev.at | 49 +++
> > 2 files changed, 84 insertions(+), 13 deletions(-)
> >
> > diff --git a/ofproto/ofproto-dpif-upcall.c
> b/ofproto/ofproto-dpif-upcall.c
> > index cd57fdbd9..5a75d9444 100644
> > --- a/ofproto/ofproto-dpif-upcall.c
> > +++ b/ofproto/ofproto-dpif-upcall.c
> > @@ -258,6 +258,7 @@ enum ukey_state {
> > UKEY_CREATED = 0,
> > UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is
> queued. */
> > UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed.
> */
> > +UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is
> inconsistent. */
> > UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is
> queued. */
> > UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
> > UKEY_DELETED, /* Ukey removed from umap, ukey free is
> deferred. */
> > @@ -1999,6 +2000,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
> ukey_state dst,
> > * UKEY_VISIBLE -> UKEY_EVICTED
> > * A handler attempts to install the flow, but the datapath
> rejects it.
> > * Consider that the datapath has already destroyed it.
> > + * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
> > + * A revalidator modifies the flow with error returns.
> > + * UKEY_INCONSISTENT -> UKEY_EVICTING
> > + * A revalidator decides to evict the datapath flow.
> > * UKEY_OPERATIONAL -> UKEY_EVICTING
> > * A revalidator decides to evict the datapath flow.
> > * UKEY_EVICTING-> UKEY_EVICTED
> > @@ -2007,7 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
> ukey_state dst,
> > * A revalidator has removed the ukey from the umap and is
> deleting it.
> > */
> > if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
> > - dst < UKEY_DELETED)) {
> > + dst < UKEY_DELETED) ||
> > + (ukey->state == UKEY_OPERATIONAL &&
>
> Indentation is wrong here, but maybe we should fix both lines, so it’s
> easier to read:
>
> if (ukey->state == dst - 1 ||
> (ukey->state == UKEY_VISIBLE && dst < UKEY_DELETED) ||
> (ukey->state == UKEY_OPERATIONAL && dst == UKEY_EVICTING)) {
>
>
>
> > + dst == UKEY_EVICTING)) {
> > ukey->state = dst;
> > } else {
> > struct ds ds = DS_EMPTY_INITIALIZER;
> > @@ -2472,26 +2479,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op
> *ops, size_t n_ops)
> >
> > for (i = 0; i < n_ops; i++) {
> > struct ukey_op *op = [i];
> > -struct dpif_flow_stats *push, *stats, push_buf;
> > -
> > -stats = op->dop.flow_del.stats;
> > -push = _buf;
> > -
> > -if (op->dop.type != DPIF_OP_FLOW_DEL) {
> > -/* Only deleted flows need their stats pushed. */
> > -continue;
> > -}
> >
> > if (op->dop.error) {
> > -/* flow_del error, 'stats' is unusable. */
> > if (op->ukey) {
> > ovs_mutex_lock(>ukey->mutex);
> > -transition_ukey(op->ukey, UKEY_EVICTED);
> > +
Re: [ovs-dev] [PATCH] ofproto-dpif-xlate: do not unwildcard source address if not needed
Ok, no problem Simon Horman 于2023年6月8日 周四22:51写道： > On Thu, Jun 08, 2023 at 10:32:45PM +0800, Peng He wrote: > > This patch has a bug, it needs 2 patches to realize the purpose in fact. > I > > can rebase if needed > > I am unclear on any need. > Perhaps it's best to drop this for now. > And revisit if the need arises. > -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] dpif-netdev: fix dpif_netdev_flow_put
Thanks for the review, will send a v2
Eelco Chaudron 于2023年6月8日 周四21:48写道：
>
>
> On 2 Jun 2023, at 20:42, Peng He wrote:
>
> > OVS allows overlapping megaflows, as long as the actions of these
> > megaflows are equal. However, the current implementation of action
> > modification relies on flow_lookup instead of ufid, this could result
> > in looking up a wrong megaflow and make the ukeys and megaflows
> inconsistent
> >
> > Just like the test case in the patch, at first we have a rule with the
> > prefix:
> >
> > 10.1.2.0/24
> >
> > and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with
> IP
> > 10.1.2.2 is received.
> >
> > Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep
> the
> > 10.1.2.2/24 megaflow and just changes its action instead of extending
> > the prefix into 10.1.2.2/16.
> >
> > then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
> > this time, we will have an overlapping megaflow with the right prefix:
> > 10.1.0.2/16
> >
> > now we have two megaflows:
> > 10.1.2.2/24
> > 10.1.0.2/16
> >
> > last, suppose we have changed the ruleset again. The revalidator this
> > time still decides to change the actions of both megaflows instead of
> > deleting them.
> >
> > The dpif_netdev_flow_put will search the megaflow to modify with unmasked
> > keys, however it might lookup the wrong megaflow as the key 10.1.2.2
> matches
> > both 10.1.2.2/24 and 10.1.0.2/16!
> >
> > This patch changes the megaflow lookup code in modification path into
> > relying the ufid to find the correct megaflow instead of key lookup.
> >
> > Signed-off-by: Peng He
>
> Hi Peng,
>
> This patch looks good in general, some style comments, and a problem with
> the tests.
>
> Thanks,
>
> Eelco
>
> > ---
> > lib/dpif-netdev.c | 64 ++-
> > tests/pmd.at | 55
> > 2 files changed, 90 insertions(+), 29 deletions(-)
> >
> > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> > index 70b953ae6..1362f27e6 100644
> > --- a/lib/dpif-netdev.c
> > +++ b/lib/dpif-netdev.c
> > @@ -4198,36 +4198,43 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
> > memset(stats, 0, sizeof *stats);
> > }
> >
> > -ovs_mutex_lock(>flow_mutex);
> > -netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > -if (!netdev_flow) {
> > -if (put->flags & DPIF_FP_CREATE) {
> > +if (put->flags & DPIF_FP_CREATE) {
> > +ovs_mutex_lock(>flow_mutex);
> > +netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > +if (!netdev_flow) {
> > dp_netdev_flow_add(pmd, match, ufid, put->actions,
> > - put->actions_len, ODPP_NONE);
> > +put->actions_len, ODPP_NONE);
> > } else {
> > -error = ENOENT;
> > +error = EEXIST;
> > }
> > +ovs_mutex_unlock(>flow_mutex);
> > } else {
> > +netdev_flow = dp_netdev_pmd_find_flow(pmd, ufid,
> > + put->key, put->key_len);
> > +
> > if (put->flags & DPIF_FP_MODIFY) {
> > -struct dp_netdev_actions *new_actions;
> > -struct dp_netdev_actions *old_actions;
> > +if (!netdev_flow) {
> > +error = ENOENT;
> > +} else {
> > +struct dp_netdev_actions *new_actions;
> > +struct dp_netdev_actions *old_actions;
> >
> > -new_actions = dp_netdev_actions_create(put->actions,
> > - put->actions_len);
> > +new_actions = dp_netdev_actions_create(put->actions,
> > +put->actions_len);
>
> Indentation is off here, and below.
>
> >
> > -old_actions = dp_netdev_flow_get_actions(netdev_flow);
> > -ovsrcu_set(_flow->actions, new_actions);
> > +old_actions = dp_netdev_flow_get_actions(netdev_flow);
> > +ovsrcu_set(_flow->actions, new_actions);
> >
> > -queue_netdev_flow_put(pmd, netdev_flow, match,
> > - put->actions, put->actions_len,
> > - DP_NETDEV_FLOW_OFFLOAD_OP_MOD);
> >
Re: [ovs-dev] [ovs-dev v2] dpif-netdev: change execute->flow->in_port into odp_port
If ipf ctx is needed this one is needed too, if not, you can drop the patch Simon Horman 于2023年6月8日 周四22:16写道： > On Thu, Mar 10, 2022 at 04:41:15AM +0000, Peng He wrote: > > the flow->in_port value is previously ignored. We normally use > > the the md->in_port of the first packet in the batch to get the > > value. However, since some actions might need to be redo due to > > the fix of ipf issues, we might have an empty batch to feed the > > action execute, so the conntrack makes use of flow->in_port > > to get the in_port value and form the ipf_ctx. > > > > This patch changes execute->flow->in_port into odp_port before > > doing action executes. with this patch, the ipf_ctx patch will > > not cause testsuite report errors. > > > > Signed-off-by: Peng He > > Acked-by: Mike Pattrick > > Signed-off-by: Aaron Conole > > > > v1->v2: change in_port in dpif layer rather than ofproto-dpif layer > > Hi, > > I noticed this stale patch in patchwork. > > It does still apply. But does not seem to have got any attention. > Moreover the patch robot flagged a problem with the sign-off from Peng He. > > If it is still relevant please consider (rebasing?) and reposting. > I am marking this as changes requested in patchwork. > -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH] ofproto-dpif-xlate: do not unwildcard source address if not needed
This patch has a bug, it needs 2 patches to realize the purpose in fact. I can rebase if needed Simon Horman 于2023年6月8日 周四22:19写道： > On Sun, May 17, 2020 at 01:08:47PM +0800, hepeng.0320 wrote: > > From: hepeng > > > > if the tunnel is specified as "remote_ip=flow", we can try to generate a > megaflow > > which ignores all source address, such as source mac, source IP address > > in the outter header of the packet. This can reduce the number of > > megaflows and avoid expensive upcall and improve the performance. > > > > Signed-off-by: hepeng > > Hi, > > I noticed this patch while looking at patchwork. > It seems to be stale and no longer applies. > > If it is still relevant please consider rebasing and reposting. > I am marking this version as "Not Applicable" in patchwork. > > -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v10] ofproto-dpif-upcall: fix push_dp_ops
push_dp_ops only handles delete ops errors but ignores the modify
ops results. It's better to handle all the dp operation errors in
a consistent way.
This patch prevents the inconsistency by considering modify failure
in revalidators.
To note, we cannot perform two state transitions and change ukey_state
into UKEY_EVICTED directly here, because, if we do so, the
sweep will remove the ukey alone and leave dp flow alive. Later, the
dump will retrieve the dp flow and might even recover it. This will
contribute the stats of this dp flow twice.
v8->v9: add testsuite and delete INCONSISTENT ukey at revalidate_sweep.
v9->v10: change the commit message and refine the test case.
Signed-off-by: Peng He
---
ofproto/ofproto-dpif-upcall.c | 48 --
tests/dpif-netdev.at | 49 +++
2 files changed, 84 insertions(+), 13 deletions(-)
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index cd57fdbd9..5a75d9444 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -258,6 +258,7 @@ enum ukey_state {
UKEY_CREATED = 0,
UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is queued. */
UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed. */
+UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is inconsistent. */
UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is queued. */
UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
UKEY_DELETED, /* Ukey removed from umap, ukey free is deferred. */
@@ -1999,6 +2000,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_VISIBLE -> UKEY_EVICTED
* A handler attempts to install the flow, but the datapath rejects it.
* Consider that the datapath has already destroyed it.
+ * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
+ * A revalidator modifies the flow with error returns.
+ * UKEY_INCONSISTENT -> UKEY_EVICTING
+ * A revalidator decides to evict the datapath flow.
* UKEY_OPERATIONAL -> UKEY_EVICTING
* A revalidator decides to evict the datapath flow.
* UKEY_EVICTING-> UKEY_EVICTED
@@ -2007,7 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* A revalidator has removed the ukey from the umap and is deleting it.
*/
if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
- dst < UKEY_DELETED)) {
+ dst < UKEY_DELETED) ||
+ (ukey->state == UKEY_OPERATIONAL &&
+ dst == UKEY_EVICTING)) {
ukey->state = dst;
} else {
struct ds ds = DS_EMPTY_INITIALIZER;
@@ -2472,26 +2479,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops,
size_t n_ops)
for (i = 0; i < n_ops; i++) {
struct ukey_op *op = [i];
-struct dpif_flow_stats *push, *stats, push_buf;
-
-stats = op->dop.flow_del.stats;
-push = _buf;
-
-if (op->dop.type != DPIF_OP_FLOW_DEL) {
-/* Only deleted flows need their stats pushed. */
-continue;
-}
if (op->dop.error) {
-/* flow_del error, 'stats' is unusable. */
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
-transition_ukey(op->ukey, UKEY_EVICTED);
+if (op->ukey->state == UKEY_EVICTING) {
+transition_ukey(op->ukey, UKEY_EVICTED);
+} else if (op->ukey->state == UKEY_OPERATIONAL) {
+/* Modify failed, ukey's state was UKEY_OPERATIONAL */
+transition_ukey(op->ukey, UKEY_INCONSISTENT);
+}
ovs_mutex_unlock(>ukey->mutex);
}
continue;
}
+if (op->dop.type != DPIF_OP_FLOW_DEL) {
+/* Only deleted flows need their stats pushed. */
+continue;
+}
+
+struct dpif_flow_stats *push, *stats, push_buf;
+
+stats = op->dop.flow_del.stats;
+push = _buf;
+
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
transition_ukey(op->ukey, UKEY_EVICTED);
@@ -2839,6 +2851,15 @@ revalidate(struct revalidator *revalidator)
continue;
}
+if (ukey->state == UKEY_INCONSISTENT) {
+ukey->dump_seq = dump_seq;
+reval_op_init([n_ops++], UKEY_DELETE, udpif, ukey,
+ , _actions);
+ovs_mutex_unlock(>mutex);
+continue;
+}
+
+
if (ukey->state <= UKEY_OPERATIONAL) {
/* The flow is now confirmed to be in the datapath. */
Re: [ovs-dev] [ovs-dev v9] ofproto-dpif-upcall: fix push_dp_ops
Recently I have identified the real root cause for this inconsistency between ukey and megaflow actions. I thus decided to withdraw this version and change its commits. Will send a v10. Please do not review this version. 0-day Robot 于2023年6月2日周五 23:19写道： > Bleep bloop. Greetings Peng He, I am a robot and I have tried out your > patch. > Thanks for your contribution. > > I encountered some error that I wasn't expecting. See the details below. > > > checkpatch: > ERROR: Author Peng He needs to sign off. > WARNING: Unexpected sign-offs from developers who are not authors or > co-authors or committers: Peng He > Lines checked: 207, Warnings: 1, Errors: 1 > > > Please check this out. If you feel there has been an error, please email > acon...@redhat.com > > Thanks, > 0-day Robot > -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] dpif-netdev: fix dpif_netdev_flow_put
OVS allows overlapping megaflows, as long as the actions of these
megaflows are equal. However, the current implementation of action
modification relies on flow_lookup instead of ufid, this could result
in looking up a wrong megaflow and make the ukeys and megaflows inconsistent
Just like the test case in the patch, at first we have a rule with the
prefix:
10.1.2.0/24
and we will get a megaflow with prefixes 10.1.2.2/24 when a packet with IP
10.1.2.2 is received.
Then suppose we change the rule into 10.1.0.0/16. OVS prefers to keep the
10.1.2.2/24 megaflow and just changes its action instead of extending
the prefix into 10.1.2.2/16.
then suppose we have a 10.1.0.2 packet, since it misses the megaflow,
this time, we will have an overlapping megaflow with the right prefix:
10.1.0.2/16
now we have two megaflows:
10.1.2.2/24
10.1.0.2/16
last, suppose we have changed the ruleset again. The revalidator this
time still decides to change the actions of both megaflows instead of
deleting them.
The dpif_netdev_flow_put will search the megaflow to modify with unmasked
keys, however it might lookup the wrong megaflow as the key 10.1.2.2 matches
both 10.1.2.2/24 and 10.1.0.2/16!
This patch changes the megaflow lookup code in modification path into
relying the ufid to find the correct megaflow instead of key lookup.
Signed-off-by: Peng He
---
lib/dpif-netdev.c | 64 ++-
tests/pmd.at | 55
2 files changed, 90 insertions(+), 29 deletions(-)
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index 70b953ae6..1362f27e6 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -4198,36 +4198,43 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
memset(stats, 0, sizeof *stats);
}
-ovs_mutex_lock(>flow_mutex);
-netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
-if (!netdev_flow) {
-if (put->flags & DPIF_FP_CREATE) {
+if (put->flags & DPIF_FP_CREATE) {
+ovs_mutex_lock(>flow_mutex);
+netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
+if (!netdev_flow) {
dp_netdev_flow_add(pmd, match, ufid, put->actions,
- put->actions_len, ODPP_NONE);
+put->actions_len, ODPP_NONE);
} else {
-error = ENOENT;
+error = EEXIST;
}
+ovs_mutex_unlock(>flow_mutex);
} else {
+netdev_flow = dp_netdev_pmd_find_flow(pmd, ufid,
+ put->key, put->key_len);
+
if (put->flags & DPIF_FP_MODIFY) {
-struct dp_netdev_actions *new_actions;
-struct dp_netdev_actions *old_actions;
+if (!netdev_flow) {
+error = ENOENT;
+} else {
+struct dp_netdev_actions *new_actions;
+struct dp_netdev_actions *old_actions;
-new_actions = dp_netdev_actions_create(put->actions,
- put->actions_len);
+new_actions = dp_netdev_actions_create(put->actions,
+put->actions_len);
-old_actions = dp_netdev_flow_get_actions(netdev_flow);
-ovsrcu_set(_flow->actions, new_actions);
+old_actions = dp_netdev_flow_get_actions(netdev_flow);
+ovsrcu_set(_flow->actions, new_actions);
-queue_netdev_flow_put(pmd, netdev_flow, match,
- put->actions, put->actions_len,
- DP_NETDEV_FLOW_OFFLOAD_OP_MOD);
-log_netdev_flow_change(netdev_flow, match, old_actions,
- put->actions, put->actions_len);
+queue_netdev_flow_put(pmd, netdev_flow, match,
+put->actions, put->actions_len,
+DP_NETDEV_FLOW_OFFLOAD_OP_MOD);
+log_netdev_flow_change(netdev_flow, match, old_actions,
+put->actions, put->actions_len);
-if (stats) {
-get_dpif_flow_status(pmd->dp, netdev_flow, stats, NULL);
-}
-if (put->flags & DPIF_FP_ZERO_STATS) {
+if (stats) {
+get_dpif_flow_status(pmd->dp, netdev_flow, stats, NULL);
+}
+if (put->flags & DPIF_FP_ZERO_STATS) {
/* XXX: The userspace datapath uses thread local statistics
* (for flows), which should be updated only by the owning
* thread. Since we cannot write on stats memory here,
@@ -4237,18 +4244,17 @@ flow_put_on_pmd(struct dp_netdev_pmd_thread *pmd,
* - Should the need arise, this operation can be implemented
* by keeping a base value
Re: [ovs-dev] [ovs-dev v7 1/3] ofproto-dpif-upcall: fix push_dp_ops
Hi, Eelco and Ilya,
I have sent a new version of this patch with a test case.
Eelco Chaudron 于2023年5月15日周一 15:45写道：
>
>
> On 12 May 2023, at 3:57, Peng He wrote:
>
> > Hi,
> >
> >
> > Eelco Chaudron 于2023年5月11日周四 15:04写道：
> >
> >>
> >>
> >> On 4 May 2023, at 9:50, Peng He wrote:
> >>
> >>> Hi,
> >>>
> >>> sorry for the late reply.
> >>>
> >>> Yes, basically this means going back to v5, but with a minor
> difference.
> >>>
> >>> In the original v5, the INCONSISTENT to EVICTING change is in the
> >>> revalidate_sweep__ phrase.
> >>>
> >>> However，since you have spot that doing so in sweep phrase has a risk:
> If
> >> in
> >>> sweep phrase,
> >>> we initial a dp op of UKEY_MODIFY but fail, we will never have another
> >>> sweep phrase after
> >>> this dp_ops and eventually when doing the next round revalidator(), we
> >> have
> >>> a warning:
> >>>
> >>> VLOG_INFO("Unexpected ukey transition from state %d "
> >>> "(last transitioned from thread %u at %s)",
> >>>
> >>> Here we will put this change in flow dump phrase. When dump a megaflow,
> >> and
> >>> found
> >>> its ukey->state == INCONSISTENT, initial a UKEY delete op.
> >>
> >> I see you already sent out a v8, and I need to re-sync with all the past
> >> approaches and changes.
> >>
> >> Will try to do a “reset” and review it from scratch next week if I find
> >> some time.
> >>
> >> I guess you still do not have a way to replicate this without code
> changes.
> >>
> >
> > are you thinking to add a testcase for this ?
>
> Yes, it would be nice if we had a way to replicate this for verification,
> and even better if we could do this with a test case.
>
> //Eelco
>
> >
> >>
> >> Cheers,
> >>
> >> Eelco
> >>
> >>
> >
> > --
> > hepeng
>
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v9] ofproto-dpif-upcall: fix push_dp_ops
push_dp_ops only handles delete ops errors but ignores the modify
ops results. It's better to handle all the dp operation errors in
a consistent way.
We observe in the production environment that sometimes a megaflow
with wrong actions keep staying in datapath. The coverage command shows
revalidators have dumped several times, however the correct
actions are not set. This implies that the ukey's action does not
equal to the meagaflow's, i.e. revalidators think the underlying
megaflow's actions are correct however they are not.
We also check the megaflow using the ofproto/trace command, and the
actions are not matched with the ones in the actual magaflow. By
performing a revalidator/purge command, the right actions are set.
This patch prevents the inconsistency by considering modify failure
in revalidators.
To note, we cannot perform two state transitions and change ukey_state
into UKEY_EVICTED directly here, because, if we do so, the
sweep will remove the ukey alone and leave dp flow alive. Later, the
dump will retrieve the dp flow and might even recover it. This will
contribute the stats of this dp flow twice.
v8->v9: add testsuite and delete INCONSISTENT ukey at revalidate_sweep.
Signed-off-by: Peng He
---
ofproto/ofproto-dpif-upcall.c | 48 --
tests/dpif-netdev.at | 49 +++
2 files changed, 84 insertions(+), 13 deletions(-)
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index cd57fdbd9..5a75d9444 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -258,6 +258,7 @@ enum ukey_state {
UKEY_CREATED = 0,
UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is queued. */
UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed. */
+UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is inconsistent. */
UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is queued. */
UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
UKEY_DELETED, /* Ukey removed from umap, ukey free is deferred. */
@@ -1999,6 +2000,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_VISIBLE -> UKEY_EVICTED
* A handler attempts to install the flow, but the datapath rejects it.
* Consider that the datapath has already destroyed it.
+ * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
+ * A revalidator modifies the flow with error returns.
+ * UKEY_INCONSISTENT -> UKEY_EVICTING
+ * A revalidator decides to evict the datapath flow.
* UKEY_OPERATIONAL -> UKEY_EVICTING
* A revalidator decides to evict the datapath flow.
* UKEY_EVICTING-> UKEY_EVICTED
@@ -2007,7 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* A revalidator has removed the ukey from the umap and is deleting it.
*/
if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
- dst < UKEY_DELETED)) {
+ dst < UKEY_DELETED) ||
+ (ukey->state == UKEY_OPERATIONAL &&
+ dst == UKEY_EVICTING)) {
ukey->state = dst;
} else {
struct ds ds = DS_EMPTY_INITIALIZER;
@@ -2472,26 +2479,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops,
size_t n_ops)
for (i = 0; i < n_ops; i++) {
struct ukey_op *op = [i];
-struct dpif_flow_stats *push, *stats, push_buf;
-
-stats = op->dop.flow_del.stats;
-push = _buf;
-
-if (op->dop.type != DPIF_OP_FLOW_DEL) {
-/* Only deleted flows need their stats pushed. */
-continue;
-}
if (op->dop.error) {
-/* flow_del error, 'stats' is unusable. */
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
-transition_ukey(op->ukey, UKEY_EVICTED);
+if (op->ukey->state == UKEY_EVICTING) {
+transition_ukey(op->ukey, UKEY_EVICTED);
+} else if (op->ukey->state == UKEY_OPERATIONAL) {
+/* Modify failed, ukey's state was UKEY_OPERATIONAL */
+transition_ukey(op->ukey, UKEY_INCONSISTENT);
+}
ovs_mutex_unlock(>ukey->mutex);
}
continue;
}
+if (op->dop.type != DPIF_OP_FLOW_DEL) {
+/* Only deleted flows need their stats pushed. */
+continue;
+}
+
+struct dpif_flow_stats *push, *stats, push_buf;
+
+stats = op->dop.flow_del.stats;
+push = _buf;
+
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
transition_ukey(op->ukey, UKEY_EVICTED);
@@ -2839,6 +2851,15 @@ revalidate(struct r
Re: [ovs-dev] [PATCH v3] dpif-netdev: Use unmasked key when adding datapath flows.
Hi, Eelco,
Did this patch only fix the dummy-datapath?
since in the real world scenario, revalidator will only modify or delete
ukey, it will not add a megaflow into the datapath.
If we have a wider new key and narrowed old key, the reavlidator will
modify the action of the old key, and upcall_cb
will install the wider new key (without masking the key with the wider
mask, so in fact, we do not have the problem
you described here?)
Eelco Chaudron 于2022年11月28日周一 16:53写道：
> The datapath supports installing wider flows, and OVS relies on
> this behavior. For example if ipv4(src=1.1.1.1/192.0.0.0,
> dst=1.1.1.2/192.0.0.0) exists, a wider flow (smaller mask) of
> ipv4(src=192.1.1.1/128.0.0.0,dst=192.1.1.2/128.0.0.0) is allowed
> to be added.
>
> However, if we try to add a wildcard rule, the installation fails:
>
> # ovs-appctl dpctl/add-flow system@myDP "in_port(1),eth_type(0x0800), \
> ipv4(src=1.1.1.1/192.0.0.0,dst=1.1.1.2/192.0.0.0,frag=no)" 2
> # ovs-appctl dpctl/add-flow system@myDP "in_port(1),eth_type(0x0800), \
> ipv4(src=192.1.1.1/0.0.0.0,dst=49.1.1.2/0.0.0.0,frag=no)" 2
> ovs-vswitchd: updating flow table (File exists)
>
> The reason is that the key used to determine if the flow is already
> present in the system uses the original key ANDed with the mask.
> This results in the IP address not being part of the (miniflow) key,
> i.e., being substituted with an all-zero value. When doing the actual
> lookup, this results in the key wrongfully matching the first flow,
> and therefore the flow does not get installed. The solution is to use
> the unmasked key for the existence check, the same way this is handled
> in the "slow" dpif_flow_put() case.
>
> OVS relies on the fact that overlapping flows can exist if one is a
> superset of the other. Note that this is only true when the same set
> of actions is applied. This is due to how the revalidator process
> works. During revalidation, OVS removes too generic flows from the
> datapath to avoid incorrect matches but allows too narrow flows to
> stay in the datapath to avoid the data plane disruption and also to
> avoid constant flow deletions if the datapath ignores wildcards on
> certain fields/bits. See flow_wildcards_has_extra() check in the
> revalidate_ukey__() function.
>
> The problem here is that we have a too narrow flow installed, and now
> OpenFlow rules got changed, so the actual flow should be more generic.
> Revalidators will not remove the narrow flow, and we will eventually get
> an upcall on the packet that doesn't match the narrow flow, but we will
> not be able to install a more generic flow because after masking with
> the new wider mask, the key matches on the narrow flow, so we get EEXIST.
>
> Fixes: beb75a40fdc2 ("userspace: Switching of L3 packets in L2 pipeline")
> Signed-off-by: Eelco Chaudron
>
> ---
> v3:
> - Fixed comment text.
> v2:
> - Updated commit message and added fixes tag.
>
> lib/dpif-netdev.c| 33 +
> tests/dpif-netdev.at | 14 ++
> 2 files changed, 43 insertions(+), 4 deletions(-)
>
> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> index a45b46014..be2041ce2 100644
> --- a/lib/dpif-netdev.c
> +++ b/lib/dpif-netdev.c
> @@ -3321,6 +3321,28 @@ netdev_flow_key_init_masked(struct netdev_flow_key
> *dst,
> (dst_u64 - miniflow_get_values(>mf)) *
> 8);
> }
>
> +/* Initializes 'key' as a copy of 'flow'. */
> +static inline void
> +netdev_flow_key_init(struct netdev_flow_key *key,
> + const struct flow *flow)
> +{
> +uint64_t *dst = miniflow_values(>mf);
> +uint32_t hash = 0;
> +uint64_t value;
> +
> +miniflow_map_init(>mf, flow);
> +miniflow_init(>mf, flow);
> +
> +size_t n = dst - miniflow_get_values(>mf);
> +
> +FLOW_FOR_EACH_IN_MAPS (value, flow, key->mf.map) {
> +hash = hash_add64(hash, value);
> +}
> +
> +key->hash = hash_finish(hash, n * 8);
> +key->len = netdev_flow_key_size(n);
> +}
> +
> static inline void
> emc_change_entry(struct emc_entry *ce, struct dp_netdev_flow *flow,
> const struct netdev_flow_key *key)
> @@ -4195,7 +4217,7 @@ static int
> dpif_netdev_flow_put(struct dpif *dpif, const struct dpif_flow_put *put)
> {
> struct dp_netdev *dp = get_dp_netdev(dpif);
> -struct netdev_flow_key key, mask;
> +struct netdev_flow_key key;
> struct dp_netdev_pmd_thread *pmd;
> struct match match;
> ovs_u128 ufid;
> @@ -4244,9 +4266,12 @@ dpif_netdev_flow_put(struct dpif *dpif, const
> struct dpif_flow_put *put)
>
> /* Must produce a netdev_flow_key for lookup.
> * Use the same method as employed to create the key when adding
> - * the flow to the dplcs to make sure they match. */
> -netdev_flow_mask_init(, );
> -netdev_flow_key_init_masked(, , );
> + * the flow to the dplcs to make sure they match.
> + * We need to put in the unmasked key as flow_put_on_pmd() will first
>
Re: [ovs-dev] [PATCH 2/2] conntrack: Release nat_conn in case both keys have the same hash.
Hi, Paolo,
IIRC, you have a revision version on these patches?
I guess it should be closer to the upstream than mine?
Aaron Conole 于2023年5月17日周三 21:54写道：
> Paolo Valerio writes:
>
> > Ilya Maximets writes:
> >
> >> On 5/4/23 19:21, Paolo Valerio wrote:
> >>> Ilya Maximets writes:
> >>>
> On 4/19/23 20:40, Paolo Valerio wrote:
> > During the creation of a new connection, there's a chance both key
> and
> > rev_key end up having the same hash. This is more common in the case
> > of all-zero snat with no collisions. In that case, once the
> > connection is expired, but not cleaned up, if a new packet with the
> > same 5-tuple is received, an assertion failure gets triggered in
> > conn_update_state() because of a previous failure of retrieving a
> > CT_CONN_TYPE_DEFAULT connection.
> >
> > Fix it by releasing the nat_conn during the connection creation in
> the
> > case of same hash for both key and rev_key.
>
> This sounds a bit odd. Shouldn't we treat hash collision as a normal
> case?
>
> Looking at the code, I'm assuming that the issue comes from the
> following
> part in process_one():
>
> if (OVS_LIKELY(conn)) {
> if (conn->conn_type == CT_CONN_TYPE_UN_NAT) {
> ...
> conn_key_lookup(ct, >key, hash, now, ,
> >reply);
>
> And here we get the same connection again, because the default one is
> already
> expired. Is that correct?
>
> If so, maybe we should add an extra condition to conn_key_lookup() to
> only look for DEFAULT connections instead, just for this case? Since
> we really don't want to get the UN_NAT one here.
>
> >>>
> >>> Hello Ilya,
> >>>
> >>> It's a fair point.
> >>> I initially thought about the approach you're suggesting, but I had
> some
> >>> concerns about it that I'll try to summarize below.
> >>>
> >>> For sure it would fix the issue (it could require the first patch to be
> >>> applied as well for the branches with rcu exp lists).
> >>>
> >>> Based on the current logic, new packets matching that expired
> connection
> >>> but not evicted will be marked as +inv and further packets will be
> >>> marked so for the whole sweep interval unless an exception like this
> get
> >>> added:
> >>>
> >>> uint32_t hash = conn_key_hash(>rev_key, ct->hash_basis);
> >>> /* the last flag indicates CT_CONN_TYPE_DEFAULT only */
> >>> conn_key_lookup_(ct, >key, hash, now, , >reply, true);
> >>> /* special case where there's hash collision */
> >>> if (!conn && ctx->hash != hash) {
> >>> pkt->md.ct_state |= CS_INVALID;
> >>> write_ct_md(pkt, zone, NULL, NULL, NULL);
> >>> ...
> >>> return;
> >>> }
> >>>
> >>> This would further require that subsequent lookup in the
> create_new_conn
> >>> path are restricted to CT_CONN_TYPE_DEFAULT, e.g.:
> >>>
> >>> uint32_t hash = conn_key_hash(>key, ct->hash_basis);
> >>> /* Only check for CT_CONN_TYPE_DEFAULT */
> >>> if (!conn_key_lookup_(ct, >key, hash, now, NULL, NULL, true)) {
> >>> conn = conn_not_found(ct, pkt, ctx, commit, now, nat_action_info,
> >>> helper, alg_exp, ct_alg_ctl, tp_id);
> >>> }
> >>>
> >>> otherwise we could incur in a false positive which prevent to create a
> >>> new connection.
> >>
> >> I'm not really sure if what described above is more correct way of doing
> >> things or not... Aaron, do you have opinion on this?
> >>
> >> Another thought: Can we expire the CT_CONN_TYPE_UN_NAT connection the
> >> moment DEFAULT counterpart of it expires? Or that will that be against
> >> some logic / not possible to do?
> >>
> >
> > As far as I can tell, this could not be straightforward as simply
> > marking it as expired should not be reliable (e.g. doing it from the
> > sweeper), and I guess that managing the expiration time field for the
> > nat_conn as well would require updating the nat_conn every time the
> > default one gets updated, probably making it a bit unpractical.
> >
> > Another approach would be removing the nat_conn [1] altogether.
> > The problem in this case is backporting. Some adjustments that would add
> > to the patch might be needed for older branches.
> >
> > [1]
> >
> https://patchwork.ozlabs.org/project/openvswitch/patch/20201129033255.64647-2-hepeng.0...@bytedance.com/
>
> I think that work was interesting, and maybe the best way to go
> forward. Backports would become difficult, though - agreed.
>
> >>
> >>>
> Best regards, Ilya Maximets.
>
> >
> > Reported-by: Michael Plato
> > Fixes: 61e48c2d1db2 ("conntrack: Handle SNAT with all-zero IP
> address.")
> > Signed-off-by: Paolo Valerio
> > ---
> > In this thread [0] there are some more details. A similar
> > approach here could be to avoid to add the nat_conn to the cmap and
> > letting the sweeper release the memory for nat_conn once the whole
> > connection gets freed.
> > That approach could
Re: [ovs-dev] [ovs-dev v7 1/3] ofproto-dpif-upcall: fix push_dp_ops
I see. will check if I can add something in the dummy datapath I guess.
Eelco Chaudron 于2023年5月15日周一 15:45写道：
>
>
> On 12 May 2023, at 3:57, Peng He wrote:
>
> > Hi,
> >
> >
> > Eelco Chaudron 于2023年5月11日周四 15:04写道：
> >
> >>
> >>
> >> On 4 May 2023, at 9:50, Peng He wrote:
> >>
> >>> Hi,
> >>>
> >>> sorry for the late reply.
> >>>
> >>> Yes, basically this means going back to v5, but with a minor
> difference.
> >>>
> >>> In the original v5, the INCONSISTENT to EVICTING change is in the
> >>> revalidate_sweep__ phrase.
> >>>
> >>> However，since you have spot that doing so in sweep phrase has a risk:
> If
> >> in
> >>> sweep phrase,
> >>> we initial a dp op of UKEY_MODIFY but fail, we will never have another
> >>> sweep phrase after
> >>> this dp_ops and eventually when doing the next round revalidator(), we
> >> have
> >>> a warning:
> >>>
> >>> VLOG_INFO("Unexpected ukey transition from state %d "
> >>> "(last transitioned from thread %u at %s)",
> >>>
> >>> Here we will put this change in flow dump phrase. When dump a megaflow,
> >> and
> >>> found
> >>> its ukey->state == INCONSISTENT, initial a UKEY delete op.
> >>
> >> I see you already sent out a v8, and I need to re-sync with all the past
> >> approaches and changes.
> >>
> >> Will try to do a “reset” and review it from scratch next week if I find
> >> some time.
> >>
> >> I guess you still do not have a way to replicate this without code
> changes.
> >>
> >
> > are you thinking to add a testcase for this ?
>
> Yes, it would be nice if we had a way to replicate this for verification,
> and even better if we could do this with a test case.
>
> //Eelco
>
> >
> >>
> >> Cheers,
> >>
> >> Eelco
> >>
> >>
> >
> > --
> > hepeng
>
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v7 1/3] ofproto-dpif-upcall: fix push_dp_ops
Hi,
Eelco Chaudron 于2023年5月11日周四 15:04写道：
>
>
> On 4 May 2023, at 9:50, Peng He wrote:
>
> > Hi,
> >
> > sorry for the late reply.
> >
> > Yes, basically this means going back to v5, but with a minor difference.
> >
> > In the original v5, the INCONSISTENT to EVICTING change is in the
> > revalidate_sweep__ phrase.
> >
> > However，since you have spot that doing so in sweep phrase has a risk: If
> in
> > sweep phrase,
> > we initial a dp op of UKEY_MODIFY but fail, we will never have another
> > sweep phrase after
> > this dp_ops and eventually when doing the next round revalidator(), we
> have
> > a warning:
> >
> > VLOG_INFO("Unexpected ukey transition from state %d "
> > "(last transitioned from thread %u at %s)",
> >
> > Here we will put this change in flow dump phrase. When dump a megaflow,
> and
> > found
> > its ukey->state == INCONSISTENT, initial a UKEY delete op.
>
> I see you already sent out a v8, and I need to re-sync with all the past
> approaches and changes.
>
> Will try to do a “reset” and review it from scratch next week if I find
> some time.
>
> I guess you still do not have a way to replicate this without code changes.
>
are you thinking to add a testcase for this ?
>
> Cheers,
>
> Eelco
>
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v8 2/2] dpif-netdev: fix the race comments
Great, thanks!
Eelco Chaudron 于2023年5月11日周四 15:02写道：
>
>
> On 4 May 2023, at 13:29, Peng He wrote:
>
> > The following comments (brought in at 0de8783a9):
> >
> > /* XXX: There's a race window where a flow covering this packet
> > * could have already been installed since we last did the flow
> > * lookup before upcall. This could be solved by moving the
> > * mutex lock outside the loop, but that's an awful long time
> > * to be locking revalidators out of making flow modifications. */
> >
> > is out-dated. Back at commit 0de8783a9, the classifier is per-datapath,
> > multiple PMDs share a same classifier. Since now we have changed into
> > per-PMD classifier, the lookup code only prevents from the race
> > results from megaflows installed by another non-pmd threads, through
> > either manually calling dpctl/add-flow or handler thread installing
> > megaflow with pmd-id == PMD_ID_NULL, there are no other threads which
> > would insert datapath flows.
>
> Thanks this looks good to me. Will try to review patch 1/2 next week.
>
> Acked-by: Eelco Chaudron
>
> > Signed-off-by: Peng He
> > ---
> > lib/dpif-netdev.c | 9 -
> > 1 file changed, 4 insertions(+), 5 deletions(-)
> >
> > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> > index 70b953ae6..d92ec01fb 100644
> > --- a/lib/dpif-netdev.c
> > +++ b/lib/dpif-netdev.c
> > @@ -8403,12 +8403,11 @@ handle_packet_upcall(struct dp_netdev_pmd_thread
> *pmd,
> > if (OVS_LIKELY(error != ENOSPC)) {
> > struct dp_netdev_flow *netdev_flow;
> >
> > -/* XXX: There's a race window where a flow covering this packet
> > - * could have already been installed since we last did the flow
> > - * lookup before upcall. This could be solved by moving the
> > - * mutex lock outside the loop, but that's an awful long time
> > - * to be locking revalidators out of making flow modifications.
> */
> > ovs_mutex_lock(>flow_mutex);
> > +/* Two scenarios exist where a flow could have been added while
> > + * processing the upcall:
> > + * 1) a flow was manually added through dpctl/add-flow.
> > + * 2) a handler installed a flow with pmd-id == PMD_ID_NULL. */
> > netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > if (OVS_LIKELY(!netdev_flow)) {
> > netdev_flow = dp_netdev_flow_add(pmd, , ,
> > --
> > 2.25.1
>
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v7 3/3] dpif-netdev: fix the race comments
Hi
I have sent a new version with your re-write.
Eelco Chaudron 于2022年12月8日周四 19:08写道：
>
>
> On 27 Nov 2022, at 8:28, Peng He wrote:
>
> > The following comments (brought in at 0de8783a9):
> >
> > /* XXX: There's a race window where a flow covering this packet
> > * could have already been installed since we last did the flow
> > * lookup before upcall. This could be solved by moving the
> > * mutex lock outside the loop, but that's an awful long time
> > * to be locking revalidators out of making flow modifications. */
> >
> > is out-dated. Back at commit 0de8783a9, the classifier is per-datapath,
> > multiple PMDs share a same classifier. Since now we have changed into
> > per-PMD classifier, the lookup code only prevents from the race
> > results from megaflows installed by another threads, through either
> > manually calling dpctl/add-flow or handler thread installing megaflow
> > with pmd-id == PMD_ID_NULL, there are no other threads which would
> > insert datapath flows.
> >
> > Signed-off-by: Peng He
> > ---
> > lib/dpif-netdev.c | 9 -
> > 1 file changed, 4 insertions(+), 5 deletions(-)
> >
> > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> > index a45b46014..597faa047 100644
> > --- a/lib/dpif-netdev.c
> > +++ b/lib/dpif-netdev.c
> > @@ -8297,12 +8297,11 @@ handle_packet_upcall(struct dp_netdev_pmd_thread
> *pmd,
> > if (OVS_LIKELY(error != ENOSPC)) {
> > struct dp_netdev_flow *netdev_flow;
> >
> > -/* XXX: There's a race window where a flow covering this packet
> > - * could have already been installed since we last did the flow
> > - * lookup before upcall. This could be solved by moving the
> > - * mutex lock outside the loop, but that's an awful long time
> > - * to be locking revalidators out of making flow modifications.
> */
> > ovs_mutex_lock(>flow_mutex);
> > +/* Two scenarios that race could happen:
> > + * 1) manually add megaflow through dpctl/add-flow
> > + * 2) handler installs a megaflow with pmd-id == PMD_ID_NULL
> > + */
>
> Maybe just a little re-write so it’s clear what you mean with race.
>
> /* Two scenarios exist where a flow could have been added while
> * processing the upcall:
> * 1) a flow was manually added through dpctl/add-flow.
> * 2) a handler installed a flow with pmd-id == PMD_ID_NULL. */
>
> > netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > if (OVS_LIKELY(!netdev_flow)) {
> > netdev_flow = dp_netdev_flow_add(pmd, , ,
> > --
> > 2.25.1
>
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v8 2/2] dpif-netdev: fix the race comments
The following comments (brought in at 0de8783a9):
/* XXX: There's a race window where a flow covering this packet
* could have already been installed since we last did the flow
* lookup before upcall. This could be solved by moving the
* mutex lock outside the loop, but that's an awful long time
* to be locking revalidators out of making flow modifications. */
is out-dated. Back at commit 0de8783a9, the classifier is per-datapath,
multiple PMDs share a same classifier. Since now we have changed into
per-PMD classifier, the lookup code only prevents from the race
results from megaflows installed by another non-pmd threads, through
either manually calling dpctl/add-flow or handler thread installing
megaflow with pmd-id == PMD_ID_NULL, there are no other threads which
would insert datapath flows.
Signed-off-by: Peng He
---
lib/dpif-netdev.c | 9 -
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index 70b953ae6..d92ec01fb 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -8403,12 +8403,11 @@ handle_packet_upcall(struct dp_netdev_pmd_thread *pmd,
if (OVS_LIKELY(error != ENOSPC)) {
struct dp_netdev_flow *netdev_flow;
-/* XXX: There's a race window where a flow covering this packet
- * could have already been installed since we last did the flow
- * lookup before upcall. This could be solved by moving the
- * mutex lock outside the loop, but that's an awful long time
- * to be locking revalidators out of making flow modifications. */
ovs_mutex_lock(>flow_mutex);
+/* Two scenarios exist where a flow could have been added while
+ * processing the upcall:
+ * 1) a flow was manually added through dpctl/add-flow.
+ * 2) a handler installed a flow with pmd-id == PMD_ID_NULL. */
netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
if (OVS_LIKELY(!netdev_flow)) {
netdev_flow = dp_netdev_flow_add(pmd, , ,
--
2.25.1
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v8 1/2] ofproto-dpif-upcall: fix push_dp_ops
push_dp_ops only handles delete ops errors but ignores the modify
ops results. It's better to handle all the dp operation errors in
a consistent way.
We observe in the production environment that sometimes a megaflow
with wrong actions keep staying in datapath. The coverage command shows
revalidators have dumped several times, however the correct
actions are not set. This implies that the ukey's action does not
equal to the meagaflow's, i.e. revalidators think the underlying
megaflow's actions are correct however they are not.
We also check the megaflow using the ofproto/trace command, and the
actions are not matched with the ones in the actual magaflow. By
performing a revalidator/purge command, the right actions are set.
This patch prevents the inconsistency by considering modify failure
in revalidators.
To note, we cannot perform two state transitions and change ukey_state
into UKEY_EVICTED directly here, because, if we do so, the
sweep will remove the ukey alone and leave dp flow alive. Later, the
dump will retrieve the dp flow and might even recover it. This will
contribute the stats of this dp flow twice.
Signed-off-by: Peng He
---
ofproto/ofproto-dpif-upcall.c | 45 +--
1 file changed, 33 insertions(+), 12 deletions(-)
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index cd57fdbd9..c5b01de18 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -258,6 +258,7 @@ enum ukey_state {
UKEY_CREATED = 0,
UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is queued. */
UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed. */
+UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is inconsistent. */
UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is queued. */
UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
UKEY_DELETED, /* Ukey removed from umap, ukey free is deferred. */
@@ -1999,6 +2000,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_VISIBLE -> UKEY_EVICTED
* A handler attempts to install the flow, but the datapath rejects it.
* Consider that the datapath has already destroyed it.
+ * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
+ * A revalidator modifies the flow with error returns.
+ * UKEY_INCONSISTENT -> UKEY_EVICTING
+ * A revalidator decides to evict the datapath flow.
* UKEY_OPERATIONAL -> UKEY_EVICTING
* A revalidator decides to evict the datapath flow.
* UKEY_EVICTING-> UKEY_EVICTED
@@ -2007,7 +2012,9 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* A revalidator has removed the ukey from the umap and is deleting it.
*/
if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
- dst < UKEY_DELETED)) {
+ dst < UKEY_DELETED) ||
+ (ukey->state == UKEY_OPERATIONAL &&
+ dst == UKEY_EVICTING)) {
ukey->state = dst;
} else {
struct ds ds = DS_EMPTY_INITIALIZER;
@@ -2472,26 +2479,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops,
size_t n_ops)
for (i = 0; i < n_ops; i++) {
struct ukey_op *op = [i];
-struct dpif_flow_stats *push, *stats, push_buf;
-
-stats = op->dop.flow_del.stats;
-push = _buf;
-
-if (op->dop.type != DPIF_OP_FLOW_DEL) {
-/* Only deleted flows need their stats pushed. */
-continue;
-}
if (op->dop.error) {
-/* flow_del error, 'stats' is unusable. */
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
-transition_ukey(op->ukey, UKEY_EVICTED);
+if (op->ukey->state == UKEY_EVICTING) {
+transition_ukey(op->ukey, UKEY_EVICTED);
+} else if (op->ukey->state == UKEY_OPERATIONAL) {
+/* Modify failed, ukey's state was UKEY_OPERATIONAL */
+transition_ukey(op->ukey, UKEY_INCONSISTENT);
+}
ovs_mutex_unlock(>ukey->mutex);
}
continue;
}
+if (op->dop.type != DPIF_OP_FLOW_DEL) {
+/* Only deleted flows need their stats pushed. */
+continue;
+}
+
+struct dpif_flow_stats *push, *stats, push_buf;
+
+stats = op->dop.flow_del.stats;
+push = _buf;
+
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
transition_ukey(op->ukey, UKEY_EVICTED);
@@ -2839,6 +2851,15 @@ revalidate(struct revalidator *revalidator)
continue;
}
+if (ukey->state == UKEY_INCONSIS
Re: [ovs-dev] [ovs-dev v7 1/3] ofproto-dpif-upcall: fix push_dp_ops
Hi,
sorry for the late reply.
Yes, basically this means going back to v5, but with a minor difference.
In the original v5, the INCONSISTENT to EVICTING change is in the
revalidate_sweep__ phrase.
However，since you have spot that doing so in sweep phrase has a risk: If in
sweep phrase,
we initial a dp op of UKEY_MODIFY but fail, we will never have another
sweep phrase after
this dp_ops and eventually when doing the next round revalidator(), we have
a warning:
VLOG_INFO("Unexpected ukey transition from state %d "
"(last transitioned from thread %u at %s)",
Here we will put this change in flow dump phrase. When dump a megaflow, and
found
its ukey->state == INCONSISTENT, initial a UKEY delete op.
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2 09/10] revalidator: Fix datapath statistics update.
sorry, I was wrong. the old ukey and new ukey are independent. and current code does not copy the stats, which means the stats have already been cleared. Peng He 于2023年4月27日周四 16:56写道： > Hi, Eelco and Ilya > > It has been a long time and I see there are a lot fixes on the > revdalidator's code about this statistics code. > Has this stats inconsistent issue been solved? > > I just realize that making dump seq equal in *try_ukey_replace* is not > enough. > we might need to clear the old ukey's stats. > > since we only reuse the old ukey in EVICTED states, which means we have > executed delete-megaflow-op on this ukey and > the stats info has been pushed. > > so if there are new megaflow upcalls and replaces the existing ukey's, the > ukey->stats should be cleared > to sync with the new megaflows. > > if not, when revaldiator tries to delete this ukey, the push_dp_ops will > find inconsistent stats between ukey->stats( as it's synced with > old megaflow) and new magaflow, which is just the case you describe here. > > > > > > Peng He 于2022年5月21日周六 13:02写道： > >> Hi, Eelco and Ilya, >> >> We have observed the similar case in our production environments. >> >> After digging some code, I would like a discussion: >> >> In the situation that the ovs revalidator is under high load, i.e. the >> number of megaflows are too large, >> and the revalidators would try to delete some megaflows, at the same >> time, there are traffic keeps >> generating new megaflows through upcalls. >> >> the underlying cmap of megaflows changes due to a lot of deletion and >> insertion (cmap will change its bucket number, capacity, >> and re-place the megaflows in different buckets), resulting in a same >> magaflow might be dumped twice in two revalidator threads in one dump stage. >> >> Consider a case: >> >> revalidator 1 gets the megaflow A dumped and its related ukey u(A), and >> decide to kill the megaflow A. >> After megaflow deletion, the pmd generates a new megaflow B with the same >> ufid, i.e. the same match >> and the same actions, and replace the ukey u(A) with u(B). >> >> Now revalidator 2 gets the old megaflow A again, and it will find the >> ukey u(B). >> u(B) is the new ukey, it's dump_seq is 0, and this leads megaflow A is >> viewed as megaflow B, and its >> statistics are contributed into ukey(B). >> >> The megaflow A and megaflow B is in essential the same one, the only >> difference is the statistics value. >> This mismatch of megaflow and ukey has two side effects: >> >> 1) the megaflow A's statistics have been contributed twice, leading to a >> amplified value of openflow rule's >> statistics. This is our observation in our environments. >> >> 2) revalidator 2 might also decides to kill ukey(B), but now its >> statistics equals to megaflow A, and thus >> results in a mismatch value in the email that Eelco observed. >> >> To fix the case 1), I think, during the ukey replace, the new ukey >> generated by upcall should take the dump_seq >> value of the replaced old key, and avoid the old megaflow being dumped >> and find the new key. >> >> and the case 2) is also fixed. >> >> Any thoughts? >> >> >> Eelco Chaudron 于2022年2月22日周二 16:10写道： >> >>> >>> >>> On 21 Feb 2022, at 12:29, Eelco Chaudron wrote: >>> >>> > On 17 Feb 2022, at 14:10, Ilya Maximets wrote: >>> > >>> >> On 1/31/22 11:54, Eelco Chaudron wrote: >>> >>> Make sure to only update packet and byte counters when valid, >>> >>> or else this could lead to "temporarily/occasionally" >>> >>> out-of-sync flow counters. >>> >> >>> >> There was already the same patch submitted here: >>> >> >>> https://patchwork.ozlabs.org/project/openvswitch/patch/20200602075036.78112-1-zhaozha...@163.com/ >>> >> >>> >> And I'm still not comfortable with the change, because it seems >>> >> like it only hides the underlying datapath problem. Do you know >>> >> why exactly datapath stats become lower than previously reported? >>> >> >>> >> If it's some kind of a statistics flush, that will mean that flow >>> >> statistics will not be updated until new stats will catch up to the >>> >> old value leading to the flow revalidation and incorrect flow stats >>> >> anyway. >>> > >>> > This was very hard to reproduce, on
Re: [ovs-dev] [PATCH v2 09/10] revalidator: Fix datapath statistics update.
Hi, Eelco and Ilya It has been a long time and I see there are a lot fixes on the revdalidator's code about this statistics code. Has this stats inconsistent issue been solved? I just realize that making dump seq equal in *try_ukey_replace* is not enough. we might need to clear the old ukey's stats. since we only reuse the old ukey in EVICTED states, which means we have executed delete-megaflow-op on this ukey and the stats info has been pushed. so if there are new megaflow upcalls and replaces the existing ukey's, the ukey->stats should be cleared to sync with the new megaflows. if not, when revaldiator tries to delete this ukey, the push_dp_ops will find inconsistent stats between ukey->stats( as it's synced with old megaflow) and new magaflow, which is just the case you describe here. Peng He 于2022年5月21日周六 13:02写道： > Hi, Eelco and Ilya, > > We have observed the similar case in our production environments. > > After digging some code, I would like a discussion: > > In the situation that the ovs revalidator is under high load, i.e. the > number of megaflows are too large, > and the revalidators would try to delete some megaflows, at the same time, > there are traffic keeps > generating new megaflows through upcalls. > > the underlying cmap of megaflows changes due to a lot of deletion and > insertion (cmap will change its bucket number, capacity, > and re-place the megaflows in different buckets), resulting in a same > magaflow might be dumped twice in two revalidator threads in one dump stage. > > Consider a case: > > revalidator 1 gets the megaflow A dumped and its related ukey u(A), and > decide to kill the megaflow A. > After megaflow deletion, the pmd generates a new megaflow B with the same > ufid, i.e. the same match > and the same actions, and replace the ukey u(A) with u(B). > > Now revalidator 2 gets the old megaflow A again, and it will find the ukey > u(B). > u(B) is the new ukey, it's dump_seq is 0, and this leads megaflow A is > viewed as megaflow B, and its > statistics are contributed into ukey(B). > > The megaflow A and megaflow B is in essential the same one, the only > difference is the statistics value. > This mismatch of megaflow and ukey has two side effects: > > 1) the megaflow A's statistics have been contributed twice, leading to a > amplified value of openflow rule's > statistics. This is our observation in our environments. > > 2) revalidator 2 might also decides to kill ukey(B), but now its > statistics equals to megaflow A, and thus > results in a mismatch value in the email that Eelco observed. > > To fix the case 1), I think, during the ukey replace, the new ukey > generated by upcall should take the dump_seq > value of the replaced old key, and avoid the old megaflow being dumped and > find the new key. > > and the case 2) is also fixed. > > Any thoughts? > > > Eelco Chaudron 于2022年2月22日周二 16:10写道： > >> >> >> On 21 Feb 2022, at 12:29, Eelco Chaudron wrote: >> >> > On 17 Feb 2022, at 14:10, Ilya Maximets wrote: >> > >> >> On 1/31/22 11:54, Eelco Chaudron wrote: >> >>> Make sure to only update packet and byte counters when valid, >> >>> or else this could lead to "temporarily/occasionally" >> >>> out-of-sync flow counters. >> >> >> >> There was already the same patch submitted here: >> >> >> https://patchwork.ozlabs.org/project/openvswitch/patch/20200602075036.78112-1-zhaozha...@163.com/ >> >> >> >> And I'm still not comfortable with the change, because it seems >> >> like it only hides the underlying datapath problem. Do you know >> >> why exactly datapath stats become lower than previously reported? >> >> >> >> If it's some kind of a statistics flush, that will mean that flow >> >> statistics will not be updated until new stats will catch up to the >> >> old value leading to the flow revalidation and incorrect flow stats >> >> anyway. >> > >> > This was very hard to reproduce, only once out of 20-30 runs if I >> remember correctly. >> > >> > Without the patch, it would sometimes show very high numbers and then >> got updated after a while with the correct numbers. >> > At least this is what I remember, as I did not take any notes, and my >> brain wanted to forget this patchset :) >> > >> > >> > Guess the fix is needed anyway as this behavior was there since day one >> in revalidate_ukey(), e79a6c833. >> > >> > I also see that you got no reply to your comments, so I’ll take another >> stab to make sure this patch is real
Re: [ovs-dev] [ovs-dev v7 1/3] ofproto-dpif-upcall: fix push_dp_ops
Eelco Chaudron 于2022年12月16日周五 23:00写道： > > > On 16 Dec 2022, at 8:56, Peng He wrote: > > > From: Peng He > > To: Eelco Chaudron > > Cc: Ilya Maximets , ovs-dev@openvswitch.org > > Subject: Re: [ovs-dev v7 1/3] ofproto-dpif-upcall: fix push_dp_ops > > Date: Fri, 16 Dec 2022 15:56:32 +0800 > > > > Eelco Chaudron 于2022年12月13日周二 20:36写道： > > > >> > >> > >> On 10 Dec 2022, at 1:37, Peng He wrote: > >> > >>> Patch v5 has statistics issues. > >>> > >>> In order to solve this issue, we had a discussion. > >>> > >>> below is the quote of the email. > >>> > >>> ” > >>> After a second thought, I think maybe keeping INCONSISTENT just for the > >>> modify error is a better option. > >>> > >>> With current patch: > >>> 1. > >>> the modify error case: > >>> OPERATIONAL -> INCONSISTENT -> EVICTING -> EVICTED > >>> 2. > >>> the delete error case: > >>> EVICTING -> EVICTED > >>> > >>> Change both to INCONSISTENT: > >>> > >>> the modify error case: > >>> did not change. > >>> > >>> the delete error case: > >>> EVICTING -> INCONSISTENT -> EVICTED? > >>> > >>> “ > >>> > >>> And we agree to take the second solution. > >> > >> I know, but going over the state meanings again, UKEY_EVICTING means the > >> following: > >> > >> /* Ukey is in umap, datapath flow delete is queued. */ > >> > >> Which now no longer is the case, so should a new state not make more > sense? > >> > > > > Why it's no longer valid? > > > > In the patch, only modify failed ukey will be set to EVICTING, is it just > > right fit the meaning of > > EVICTING? (ukey in the umap, but delete operation is queued?) > > But it’s not as the delete operation is not queued, that is done in the > revalidator_sweep__() part. > Understand now. > > >> > >> Any one else has some input on this?? > >> > >>> Eelco Chaudron 于2022年12月8日周四 18:54写道： > >>> > >>>> > >>>> > >>>> On 27 Nov 2022, at 8:28, Peng He wrote: > >>>> > >>>>> push_dp_ops only handles delete ops errors but ignores the modify > >>>>> ops results. It's better to handle all the dp operation errors in > >>>>> a consistent way. > >>>>> > >>>>> We observe in the production environment that sometimes a megaflow > >>>>> with wrong actions keep staying in datapath. The coverage command > shows > >>>>> revalidators have dumped several times, however the correct > >>>>> actions are not set. This implies that the ukey's action does not > >>>>> equal to the meagaflow's, i.e. revalidators think the underlying > >>>>> megaflow's actions are correct however they are not. > >>>>> > >>>>> We also check the megaflow using the ofproto/trace command, and the > >>>>> actions are not matched with the ones in the actual magaflow. By > >>>>> performing a revalidator/purge command, the right actions are set. > >>>>> > >>>>> This patch prevents the inconsistency by considering modify failure > >>>>> in revalidators. > >>>>> > >>>>> To note, we cannot perform two state transitions and change > ukey_state > >>>>> into UKEY_EVICTED directly here, because, if we do so, the > >>>>> sweep will remove the ukey alone and leave dp flow alive. Later, the > >>>>> dump will retrieve the dp flow and might even recover it. This will > >>>>> contribute the stats of this dp flow twice. > >>>>> > >>>>> Signed-off-by: Peng He > >>>>> --- > >>>>> ofproto/ofproto-dpif-upcall.c | 34 > +++--- > >>>>> 1 file changed, 23 insertions(+), 11 deletions(-) > >>>>> > >>>>> diff --git a/ofproto/ofproto-dpif-upcall.c > >>>> b/ofproto/ofproto-dpif-upcall.c > >>>>> index 7ad728adf..c2cefbeb8 100644 > >>>>> --- a/ofproto/ofproto-dpif-upcall.c > >>>>> +++ b/ofproto/ofproto-dpif-upcall.c > >>>>> @@ -2416,26 +2416,30 @@ pus
Re: [ovs-dev] [ovs-dev v7 1/3] ofproto-dpif-upcall: fix push_dp_ops
Eelco Chaudron 于2022年12月13日周二 20:36写道：
>
>
> On 10 Dec 2022, at 1:37, Peng He wrote:
>
> > Patch v5 has statistics issues.
> >
> > In order to solve this issue, we had a discussion.
> >
> > below is the quote of the email.
> >
> > ”
> > After a second thought, I think maybe keeping INCONSISTENT just for the
> > modify error is a better option.
> >
> > With current patch:
> > 1.
> > the modify error case:
> > OPERATIONAL -> INCONSISTENT -> EVICTING -> EVICTED
> > 2.
> > the delete error case:
> > EVICTING -> EVICTED
> >
> > Change both to INCONSISTENT:
> >
> > the modify error case:
> > did not change.
> >
> > the delete error case:
> > EVICTING -> INCONSISTENT -> EVICTED?
> >
> > “
> >
> > And we agree to take the second solution.
>
> I know, but going over the state meanings again, UKEY_EVICTING means the
> following:
>
> /* Ukey is in umap, datapath flow delete is queued. */
>
> Which now no longer is the case, so should a new state not make more sense?
>
Why it's no longer valid?
In the patch, only modify failed ukey will be set to EVICTING, is it just
right fit the meaning of
EVICTING? (ukey in the umap, but delete operation is queued?)
>
> Any one else has some input on this??
>
> > Eelco Chaudron 于2022年12月8日周四 18:54写道：
> >
> >>
> >>
> >> On 27 Nov 2022, at 8:28, Peng He wrote:
> >>
> >>> push_dp_ops only handles delete ops errors but ignores the modify
> >>> ops results. It's better to handle all the dp operation errors in
> >>> a consistent way.
> >>>
> >>> We observe in the production environment that sometimes a megaflow
> >>> with wrong actions keep staying in datapath. The coverage command shows
> >>> revalidators have dumped several times, however the correct
> >>> actions are not set. This implies that the ukey's action does not
> >>> equal to the meagaflow's, i.e. revalidators think the underlying
> >>> megaflow's actions are correct however they are not.
> >>>
> >>> We also check the megaflow using the ofproto/trace command, and the
> >>> actions are not matched with the ones in the actual magaflow. By
> >>> performing a revalidator/purge command, the right actions are set.
> >>>
> >>> This patch prevents the inconsistency by considering modify failure
> >>> in revalidators.
> >>>
> >>> To note, we cannot perform two state transitions and change ukey_state
> >>> into UKEY_EVICTED directly here, because, if we do so, the
> >>> sweep will remove the ukey alone and leave dp flow alive. Later, the
> >>> dump will retrieve the dp flow and might even recover it. This will
> >>> contribute the stats of this dp flow twice.
> >>>
> >>> Signed-off-by: Peng He
> >>> ---
> >>> ofproto/ofproto-dpif-upcall.c | 34 +++---
> >>> 1 file changed, 23 insertions(+), 11 deletions(-)
> >>>
> >>> diff --git a/ofproto/ofproto-dpif-upcall.c
> >> b/ofproto/ofproto-dpif-upcall.c
> >>> index 7ad728adf..c2cefbeb8 100644
> >>> --- a/ofproto/ofproto-dpif-upcall.c
> >>> +++ b/ofproto/ofproto-dpif-upcall.c
> >>> @@ -2416,26 +2416,30 @@ push_dp_ops(struct udpif *udpif, struct ukey_op
> >> *ops, size_t n_ops)
> >>>
> >>> for (i = 0; i < n_ops; i++) {
> >>> struct ukey_op *op = [i];
> >>> -struct dpif_flow_stats *push, *stats, push_buf;
> >>> -
> >>> -stats = op->dop.flow_del.stats;
> >>> -push = _buf;
> >>> -
> >>> -if (op->dop.type != DPIF_OP_FLOW_DEL) {
> >>> -/* Only deleted flows need their stats pushed. */
> >>> -continue;
> >>> -}
> >>>
> >>> if (op->dop.error) {
> >>> -/* flow_del error, 'stats' is unusable. */
> >>> if (op->ukey) {
> >>> ovs_mutex_lock(>ukey->mutex);
> >>> -transition_ukey(op->ukey, UKEY_EVICTED);
> >>> +if (op->dop.type == DPIF_OP_FLOW_DEL) {
> >>> +transition_ukey(op->ukey, UKEY_EVICTED);
> >>> +} else {
>
> I think we could use a comment here to make sure why
Re: [ovs-dev] [ovs-dev v7 1/3] ofproto-dpif-upcall: fix push_dp_ops
sorry, the last email does not quote the full information.
by choosing the second solution, I mean
"
the delete error case:
EVICTING -> EVICTED
Change both to INCONSISTENT:
the modify error case:
did not change.
the delete error case:
EVICTING -> INCONSISTENT -> EVICTED?
this will make the state machine allows both INCONSISTENT -> EVICTING and
EVICTING -> INCONSISTENT transitions.
which I guess it's more confusing ...
Another solution is that, drop INCONSISTENT state, if modify fails, just
changes to EVICTING.
and let the revalidate or sweep to take care of EVICTING state ukey and
initial another dp_ops to remove it.
”
so, we choose to not to use INCONSISTENT.
Peng He 于2022年12月10日周六 08:37写道：
> Patch v5 has statistics issues.
>
> In order to solve this issue, we had a discussion.
>
> below is the quote of the email.
>
> ”
> After a second thought, I think maybe keeping INCONSISTENT just for the
> modify error is a better option.
>
> With current patch:
> 1.
> the modify error case:
> OPERATIONAL -> INCONSISTENT -> EVICTING -> EVICTED
> 2.
> the delete error case:
> EVICTING -> EVICTED
>
> Change both to INCONSISTENT:
>
> the modify error case:
> did not change.
>
> the delete error case:
> EVICTING -> INCONSISTENT -> EVICTED?
>
> “
>
> And we agree to take the second solution.
>
>
>
> Eelco Chaudron 于2022年12月8日周四 18:54写道：
>
>>
>>
>> On 27 Nov 2022, at 8:28, Peng He wrote:
>>
>> > push_dp_ops only handles delete ops errors but ignores the modify
>> > ops results. It's better to handle all the dp operation errors in
>> > a consistent way.
>> >
>> > We observe in the production environment that sometimes a megaflow
>> > with wrong actions keep staying in datapath. The coverage command shows
>> > revalidators have dumped several times, however the correct
>> > actions are not set. This implies that the ukey's action does not
>> > equal to the meagaflow's, i.e. revalidators think the underlying
>> > megaflow's actions are correct however they are not.
>> >
>> > We also check the megaflow using the ofproto/trace command, and the
>> > actions are not matched with the ones in the actual magaflow. By
>> > performing a revalidator/purge command, the right actions are set.
>> >
>> > This patch prevents the inconsistency by considering modify failure
>> > in revalidators.
>> >
>> > To note, we cannot perform two state transitions and change ukey_state
>> > into UKEY_EVICTED directly here, because, if we do so, the
>> > sweep will remove the ukey alone and leave dp flow alive. Later, the
>> > dump will retrieve the dp flow and might even recover it. This will
>> > contribute the stats of this dp flow twice.
>> >
>> > Signed-off-by: Peng He
>> > ---
>> > ofproto/ofproto-dpif-upcall.c | 34 +++---
>> > 1 file changed, 23 insertions(+), 11 deletions(-)
>> >
>> > diff --git a/ofproto/ofproto-dpif-upcall.c
>> b/ofproto/ofproto-dpif-upcall.c
>> > index 7ad728adf..c2cefbeb8 100644
>> > --- a/ofproto/ofproto-dpif-upcall.c
>> > +++ b/ofproto/ofproto-dpif-upcall.c
>> > @@ -2416,26 +2416,30 @@ push_dp_ops(struct udpif *udpif, struct ukey_op
>> *ops, size_t n_ops)
>> >
>> > for (i = 0; i < n_ops; i++) {
>> > struct ukey_op *op = [i];
>> > -struct dpif_flow_stats *push, *stats, push_buf;
>> > -
>> > -stats = op->dop.flow_del.stats;
>> > -push = _buf;
>> > -
>> > -if (op->dop.type != DPIF_OP_FLOW_DEL) {
>> > -/* Only deleted flows need their stats pushed. */
>> > -continue;
>> > -}
>> >
>> > if (op->dop.error) {
>> > -/* flow_del error, 'stats' is unusable. */
>> > if (op->ukey) {
>> > ovs_mutex_lock(>ukey->mutex);
>> > -transition_ukey(op->ukey, UKEY_EVICTED);
>> > +if (op->dop.type == DPIF_OP_FLOW_DEL) {
>> > +transition_ukey(op->ukey, UKEY_EVICTED);
>> > +} else {
>> > +transition_ukey(op->ukey, UKEY_EVICTING);
>> > +}
>> > ovs_mutex_unlock(>ukey->mutex);
>> > }
>> > continue;
>> > }
>> >
>> > +if (op->dop.type != DPIF_OP_FLO
Re: [ovs-dev] [ovs-dev v7 1/3] ofproto-dpif-upcall: fix push_dp_ops
Patch v5 has statistics issues.
In order to solve this issue, we had a discussion.
below is the quote of the email.
”
After a second thought, I think maybe keeping INCONSISTENT just for the
modify error is a better option.
With current patch:
1.
the modify error case:
OPERATIONAL -> INCONSISTENT -> EVICTING -> EVICTED
2.
the delete error case:
EVICTING -> EVICTED
Change both to INCONSISTENT:
the modify error case:
did not change.
the delete error case:
EVICTING -> INCONSISTENT -> EVICTED?
“
And we agree to take the second solution.
Eelco Chaudron 于2022年12月8日周四 18:54写道：
>
>
> On 27 Nov 2022, at 8:28, Peng He wrote:
>
> > push_dp_ops only handles delete ops errors but ignores the modify
> > ops results. It's better to handle all the dp operation errors in
> > a consistent way.
> >
> > We observe in the production environment that sometimes a megaflow
> > with wrong actions keep staying in datapath. The coverage command shows
> > revalidators have dumped several times, however the correct
> > actions are not set. This implies that the ukey's action does not
> > equal to the meagaflow's, i.e. revalidators think the underlying
> > megaflow's actions are correct however they are not.
> >
> > We also check the megaflow using the ofproto/trace command, and the
> > actions are not matched with the ones in the actual magaflow. By
> > performing a revalidator/purge command, the right actions are set.
> >
> > This patch prevents the inconsistency by considering modify failure
> > in revalidators.
> >
> > To note, we cannot perform two state transitions and change ukey_state
> > into UKEY_EVICTED directly here, because, if we do so, the
> > sweep will remove the ukey alone and leave dp flow alive. Later, the
> > dump will retrieve the dp flow and might even recover it. This will
> > contribute the stats of this dp flow twice.
> >
> > Signed-off-by: Peng He
> > ---
> > ofproto/ofproto-dpif-upcall.c | 34 +++---
> > 1 file changed, 23 insertions(+), 11 deletions(-)
> >
> > diff --git a/ofproto/ofproto-dpif-upcall.c
> b/ofproto/ofproto-dpif-upcall.c
> > index 7ad728adf..c2cefbeb8 100644
> > --- a/ofproto/ofproto-dpif-upcall.c
> > +++ b/ofproto/ofproto-dpif-upcall.c
> > @@ -2416,26 +2416,30 @@ push_dp_ops(struct udpif *udpif, struct ukey_op
> *ops, size_t n_ops)
> >
> > for (i = 0; i < n_ops; i++) {
> > struct ukey_op *op = [i];
> > -struct dpif_flow_stats *push, *stats, push_buf;
> > -
> > -stats = op->dop.flow_del.stats;
> > -push = _buf;
> > -
> > -if (op->dop.type != DPIF_OP_FLOW_DEL) {
> > -/* Only deleted flows need their stats pushed. */
> > -continue;
> > -}
> >
> > if (op->dop.error) {
> > -/* flow_del error, 'stats' is unusable. */
> > if (op->ukey) {
> > ovs_mutex_lock(>ukey->mutex);
> > -transition_ukey(op->ukey, UKEY_EVICTED);
> > +if (op->dop.type == DPIF_OP_FLOW_DEL) {
> > +transition_ukey(op->ukey, UKEY_EVICTED);
> > +} else {
> > +transition_ukey(op->ukey, UKEY_EVICTING);
> > +}
> > ovs_mutex_unlock(>ukey->mutex);
> > }
> > continue;
> > }
> >
> > +if (op->dop.type != DPIF_OP_FLOW_DEL) {
> > +/* Only deleted flows need their stats pushed. */
> > +continue;
> > +}
> > +
> > +struct dpif_flow_stats *push, *stats, push_buf;
> > +
> > +stats = op->dop.flow_del.stats;
> > +push = _buf;
> > +
> > if (op->ukey) {
> > ovs_mutex_lock(>ukey->mutex);
> > transition_ukey(op->ukey, UKEY_EVICTED);
> > @@ -2848,6 +2852,14 @@ revalidator_sweep__(struct revalidator
> *revalidator, bool purge)
> > continue;
> > }
> > ukey_state = ukey->state;
> > +
> > +if (ukey_state == UKEY_EVICTING) {
> > +/* previous modify operation fails on this ukey and
> ukey_state
> > + * is set to UKEY_EVICTING, issue a delete operation on
> this
> > + * ukey.
> > + */
> > +delete_op_init(udpif, [n_ops++], ukey);
>
> How can we be sure this state here is only for failing
[ovs-dev] [ovs-dev v7 2/3] ofproto-dpif-upcall: new ukey needs to take the old ukey's dump seq
The userspace datapath mananges all the magaflows by a cmap. The cmap data structrue will grow/shrink during the datapath processing and it will re-position megaflows. This might result in two revalidator threads might process a same megaflow during one dump stage. Consider a situation that, revalidator 1 processes a megaflow A, and decides to delete it from the datapath, at the mean time, this megaflow A is also queued in the process batch of revalidator 2. Normally it's ok for revalidators to process the same megaflow multiple times, as the dump_seq shows it's already dumped and the stats will not be contributed twice. Assume that right after A is deleted, a PMD thread generates again a new megaflow B which has the same match and action of A. The ukey of megaflow B will replace the one of megaflow A. Now the ukey B is new to the revalidator system and its dump seq is 0. Now since the dump seq of ukey B is 0, when processing megaflow A, the revalidator 2 will not identify this megaflow A has already been dumped by revalidator 1 and will contribute the old megaflow A's stats again, this results in an inconsistent stats between ukeys and megaflows. To fix this, the newly generated the ukey B should take the dump_seq of the replaced ukey A to avoid a same megaflow being revalidated twice in one dump stage. We observe in the production environment, the OpenFlow rules' stats sometimes are amplified compared to the actual value. Signed-off-by: Peng He Acked-by: Eelco Chaudron --- ofproto/ofproto-dpif-upcall.c | 1 + 1 file changed, 1 insertion(+) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index c2cefbeb8..8b5db6103 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -1877,6 +1877,7 @@ try_ukey_replace(struct umap *umap, struct udpif_key *old_ukey, ovs_mutex_lock(_ukey->mutex); cmap_replace(>cmap, _ukey->cmap_node, _ukey->cmap_node, new_ukey->hash); +new_ukey->dump_seq = old_ukey->dump_seq; ovsrcu_postpone(ukey_delete__, old_ukey); transition_ukey(old_ukey, UKEY_DELETED); transition_ukey(new_ukey, UKEY_VISIBLE); -- 2.25.1 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v7 3/3] dpif-netdev: fix the race comments
The following comments (brought in at 0de8783a9):
/* XXX: There's a race window where a flow covering this packet
* could have already been installed since we last did the flow
* lookup before upcall. This could be solved by moving the
* mutex lock outside the loop, but that's an awful long time
* to be locking revalidators out of making flow modifications. */
is out-dated. Back at commit 0de8783a9, the classifier is per-datapath,
multiple PMDs share a same classifier. Since now we have changed into
per-PMD classifier, the lookup code only prevents from the race
results from megaflows installed by another threads, through either
manually calling dpctl/add-flow or handler thread installing megaflow
with pmd-id == PMD_ID_NULL, there are no other threads which would
insert datapath flows.
Signed-off-by: Peng He
---
lib/dpif-netdev.c | 9 -
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index a45b46014..597faa047 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -8297,12 +8297,11 @@ handle_packet_upcall(struct dp_netdev_pmd_thread *pmd,
if (OVS_LIKELY(error != ENOSPC)) {
struct dp_netdev_flow *netdev_flow;
-/* XXX: There's a race window where a flow covering this packet
- * could have already been installed since we last did the flow
- * lookup before upcall. This could be solved by moving the
- * mutex lock outside the loop, but that's an awful long time
- * to be locking revalidators out of making flow modifications. */
ovs_mutex_lock(>flow_mutex);
+/* Two scenarios that race could happen:
+ * 1) manually add megaflow through dpctl/add-flow
+ * 2) handler installs a megaflow with pmd-id == PMD_ID_NULL
+ */
netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
if (OVS_LIKELY(!netdev_flow)) {
netdev_flow = dp_netdev_flow_add(pmd, , ,
--
2.25.1
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v7 1/3] ofproto-dpif-upcall: fix push_dp_ops
push_dp_ops only handles delete ops errors but ignores the modify
ops results. It's better to handle all the dp operation errors in
a consistent way.
We observe in the production environment that sometimes a megaflow
with wrong actions keep staying in datapath. The coverage command shows
revalidators have dumped several times, however the correct
actions are not set. This implies that the ukey's action does not
equal to the meagaflow's, i.e. revalidators think the underlying
megaflow's actions are correct however they are not.
We also check the megaflow using the ofproto/trace command, and the
actions are not matched with the ones in the actual magaflow. By
performing a revalidator/purge command, the right actions are set.
This patch prevents the inconsistency by considering modify failure
in revalidators.
To note, we cannot perform two state transitions and change ukey_state
into UKEY_EVICTED directly here, because, if we do so, the
sweep will remove the ukey alone and leave dp flow alive. Later, the
dump will retrieve the dp flow and might even recover it. This will
contribute the stats of this dp flow twice.
Signed-off-by: Peng He
---
ofproto/ofproto-dpif-upcall.c | 34 +++---
1 file changed, 23 insertions(+), 11 deletions(-)
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index 7ad728adf..c2cefbeb8 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -2416,26 +2416,30 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops,
size_t n_ops)
for (i = 0; i < n_ops; i++) {
struct ukey_op *op = [i];
-struct dpif_flow_stats *push, *stats, push_buf;
-
-stats = op->dop.flow_del.stats;
-push = _buf;
-
-if (op->dop.type != DPIF_OP_FLOW_DEL) {
-/* Only deleted flows need their stats pushed. */
-continue;
-}
if (op->dop.error) {
-/* flow_del error, 'stats' is unusable. */
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
-transition_ukey(op->ukey, UKEY_EVICTED);
+if (op->dop.type == DPIF_OP_FLOW_DEL) {
+transition_ukey(op->ukey, UKEY_EVICTED);
+} else {
+transition_ukey(op->ukey, UKEY_EVICTING);
+}
ovs_mutex_unlock(>ukey->mutex);
}
continue;
}
+if (op->dop.type != DPIF_OP_FLOW_DEL) {
+/* Only deleted flows need their stats pushed. */
+continue;
+}
+
+struct dpif_flow_stats *push, *stats, push_buf;
+
+stats = op->dop.flow_del.stats;
+push = _buf;
+
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
transition_ukey(op->ukey, UKEY_EVICTED);
@@ -2848,6 +2852,14 @@ revalidator_sweep__(struct revalidator *revalidator,
bool purge)
continue;
}
ukey_state = ukey->state;
+
+if (ukey_state == UKEY_EVICTING) {
+/* previous modify operation fails on this ukey and ukey_state
+ * is set to UKEY_EVICTING, issue a delete operation on this
+ * ukey.
+ */
+delete_op_init(udpif, [n_ops++], ukey);
+}
if (ukey_state == UKEY_OPERATIONAL
|| (ukey_state == UKEY_VISIBLE && purge)) {
struct recirc_refs recircs = RECIRC_REFS_EMPTY_INITIALIZER;
--
2.25.1
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v6 2/3] ofproto-dpif-upcall: new ukey needs to take the old ukey's dump seq
The userspace datapath mananges all the magaflows by a cmap. The cmap data structrue will grow/shrink during the datapath processing and it will re-position megaflows. This might result in two revalidator threads might process a same megaflow during one dump stage. Consider a situation that, revalidator 1 processes a megaflow A, and decides to delete it from the datapath, at the mean time, this megaflow A is also queued in the process batch of revalidator 2. Normally it's ok for revalidators to process the same megaflow multiple times, as the dump_seq shows it's already dumped and the stats will not be contributed twice. Assume that right after A is deleted, a PMD thread generates again a new megaflow B which has the same match and action of A. The ukey of megaflow B will replace the one of megaflow A. Now the ukey B is new to the revalidator system and its dump seq is 0. Now since the dump seq of ukey B is 0, when processing megaflow A, the revalidator 2 will not identify this megaflow A has already been dumped by revalidator 1 and will contribute the old megaflow A's stats again, this results in an inconsistent stats between ukeys and megaflows. To fix this, the newly generated the ukey B should take the dump_seq of the replaced ukey A to avoid a same megaflow being revalidated twice in one dump stage. We observe in the production environment, the OpenFlow rules' stats sometimes are amplified compared to the actual value. Signed-off-by: Peng He Acked-by: Eelco Chaudron --- ofproto/ofproto-dpif-upcall.c | 1 + 1 file changed, 1 insertion(+) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 97e081681..329a58bd5 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -1877,6 +1877,7 @@ try_ukey_replace(struct umap *umap, struct udpif_key *old_ukey, ovs_mutex_lock(_ukey->mutex); cmap_replace(>cmap, _ukey->cmap_node, _ukey->cmap_node, new_ukey->hash); +new_ukey->dump_seq = old_ukey->dump_seq; ovsrcu_postpone(ukey_delete__, old_ukey); transition_ukey(old_ukey, UKEY_DELETED); transition_ukey(new_ukey, UKEY_VISIBLE); -- 2.25.1 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v6 1/3] ofproto-dpif-upcall: fix push_dp_ops
push_dp_ops only handles delete ops errors but ignores the modify
ops results. It's better to handle all the dp operation errors in
a consistent way.
We observe in the production environment that sometimes a megaflow
with wrong actions keep staying in datapath. The coverage command shows
revalidators have dumped several times, however the correct
actions are not set. This implies that the ukey's action does not
equal to the meagaflow's, i.e. revalidators think the underlying
megaflow's actions are correct however they are not.
We also check the megaflow using the ofproto/trace command, and the
actions are not matched with the ones in the actual magaflow. By
performing a revalidator/purge command, the right actions are set.
This patch prevents the inconsistency by considering modify failure
in revalidators.
To note, we cannot perform two state transitions and change ukey_state
into UKEY_EVICTED directly here, because, if we do so, the
sweep will remove the ukey alone and leave dp flow alive. Later, the
dump will retrieve the dp flow and might even recover it. This will
contribute the stats of this dp flow twice.
Signed-off-by: Peng He
---
ofproto/ofproto-dpif-upcall.c | 33 ++---
1 file changed, 22 insertions(+), 11 deletions(-)
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index 7ad728adf..97e081681 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -2416,26 +2416,30 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops,
size_t n_ops)
for (i = 0; i < n_ops; i++) {
struct ukey_op *op = [i];
-struct dpif_flow_stats *push, *stats, push_buf;
-
-stats = op->dop.flow_del.stats;
-push = _buf;
-
-if (op->dop.type != DPIF_OP_FLOW_DEL) {
-/* Only deleted flows need their stats pushed. */
-continue;
-}
if (op->dop.error) {
-/* flow_del error, 'stats' is unusable. */
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
-transition_ukey(op->ukey, UKEY_EVICTED);
+if (op->dop.type == DPIF_OP_FLOW_DEL)
+transition_ukey(op->ukey, UKEY_EVICTED);
+else {
+transition_ukey(op->ukey, UKEY_EVICTING);
+}
ovs_mutex_unlock(>ukey->mutex);
}
continue;
}
+if (op->dop.type != DPIF_OP_FLOW_DEL) {
+/* Only deleted flows need their stats pushed. */
+continue;
+}
+
+struct dpif_flow_stats *push, *stats, push_buf;
+
+stats = op->dop.flow_del.stats;
+push = _buf;
+
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
transition_ukey(op->ukey, UKEY_EVICTED);
@@ -2848,6 +2852,13 @@ revalidator_sweep__(struct revalidator *revalidator,
bool purge)
continue;
}
ukey_state = ukey->state;
+
+if (ukey_state == UKEY_EVICTING) {
+/* previous modify operation fails on this ukey and ukey_state
is set
+ * to UKEY_EVICTING, issue a delete operation on this ukey.
+ */
+delete_op_init(udpif, [n_ops++], ukey);
+}
if (ukey_state == UKEY_OPERATIONAL
|| (ukey_state == UKEY_VISIBLE && purge)) {
struct recirc_refs recircs = RECIRC_REFS_EMPTY_INITIALIZER;
--
2.25.1
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v6 3/3] dpif-netdev: fix the race comments
The following comments (brought in at 0de8783a9):
/* XXX: There's a race window where a flow covering this packet
* could have already been installed since we last did the flow
* lookup before upcall. This could be solved by moving the
* mutex lock outside the loop, but that's an awful long time
* to be locking revalidators out of making flow modifications. */
is out-dated. Back at commit 0de8783a9, the classifier is per-datapath,
multiple PMDs share a same classifier. Since now we have changed into
per-PMD classifier, the lookup code only prevents from the race
results from megaflows installed by another threads, through either
manually calling dpctl/add-flow or handler thread installing megaflow
with pmd-id == PMD_ID_NULL, there are no other threads which would
insert datapath flows.
Signed-off-by: Peng He
---
lib/dpif-netdev.c | 9 -
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index a45b46014..597faa047 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -8297,12 +8297,11 @@ handle_packet_upcall(struct dp_netdev_pmd_thread *pmd,
if (OVS_LIKELY(error != ENOSPC)) {
struct dp_netdev_flow *netdev_flow;
-/* XXX: There's a race window where a flow covering this packet
- * could have already been installed since we last did the flow
- * lookup before upcall. This could be solved by moving the
- * mutex lock outside the loop, but that's an awful long time
- * to be locking revalidators out of making flow modifications. */
ovs_mutex_lock(>flow_mutex);
+/* Two scenarios that race could happen:
+ * 1) manually add megaflow through dpctl/add-flow
+ * 2) handler installs a megaflow with pmd-id == PMD_ID_NULL
+ */
netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
if (OVS_LIKELY(!netdev_flow)) {
netdev_flow = dp_netdev_flow_add(pmd, , ,
--
2.25.1
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v4 3/3] dpif-netdev: fix inconsistent processing between ukey and megaflow
Eelco Chaudron 于2022年11月24日周四 17:08写道： > > > On 24 Nov 2022, at 10:04, Peng He wrote: > > > Eelco Chaudron 于2022年11月24日周四 16:34写道： > > > >> > >> > >> On 24 Nov 2022, at 1:46, Peng He wrote: > >> > >>> So do we need this patch (补丁) (补丁) or not?? > >>> > >>> Guessing it's quite rare in the real production environment that we > have > >>> two datapaths at the same time > >>> And I am more (更多) (更多) curious that even though we have 2 datapaths, > should > >> the port > >>> id be different? Is one > >>> port capable of being assigned to 2 datapaths at the same time > >>> > >>> Because only when a port is assigned to 2 datapaths at the same time, > we > >>> should worry about this race > >> > >> I think we should still add this patch (补丁) (补丁) , as it’s common to > have a > >> single DPDK datapath bridge, but it could have kernel (内核) (内核) (the > bridge > >> itself for example) and DPDK ports. In this case when the actions of the > >> OpenFlow rule change there could be different actions for existing rules > >> not yet updated by the revalidator. > >> > >> Or do I miss the point here? > >> > >> On a single DPDK datapath, the kernel (内核) ports will be processed by > the DPDK > > datapath also (through using AF_SOCKET socket), so in this case, we only > > have one datapath and we will not have this race. > > If you are sure we can drop this patch (补丁) . > I am sure that normally we will have only one datapath. I am not sure that why kernel datatpath's megaflow should be added into the userspace datapath But I guess we can drop this patch. Thanks! > > >> //Eelco > >> > >>> Eelco Chaudron 于2022年11月23日周三 23:54写道： > >>> > >>>> > >>>> > >>>> On 19 Nov 2022, at 1:46, Peng He wrote: > >>>> > >>>>> Eelco Chaudron 于2022年11月18日周五 15:38写道： > >>>>> > >>>>>> > >>>>>> > >>>>>> On 18 Nov 2022, at 2:57, Peng He wrote: > >>>>>> > >>>>>>> Since there are possible race conditions (between the kernel (内核) > (内核) > >> (内核) > >>>>>> datapath and > >>>>>>> userspace datapath), > >>>>>>> I guess this patch (补丁) (补丁) (补丁) is now needed again? But two > datapath > >> is really > >>>>>> rare in > >>>>>>> the real deployment. > >>>>>>> So I am not sure if we should pay attention here. > >>>>>> > >>>>>> I still think we should add this, as there seem to be a decent > amount > >> of > >>>>>> times people intermix a kernel (内核) (内核) (内核) interface with a DPDK > one. > >> For > >>>> example, > >>>>>> the bridge interface, which would be up to get routing (溃败) (溃败) > (溃败) > >> information > >>>> for > >>>>>> tunnels. > >>>>> > >>>>> > >>>>> In this case, bridge interfaces are attached (附加) (附加) to the > userspace > >> datapath, > >>>> it > >>>>> will be " polled (民意调查) (民意调查) " by the main thread, and it's pmd-id > is > >> NON_PMD_CORE_ID. > >>>>> > >>>>> The case that race could happen is that mix using of userspace > datapath > >>>> and > >>>>> kernel (内核) (内核) datapath. When the kernel datapath receives a > upcall, it > >> will set > >>>>> the pmd-id to PMD_ID_NULL. Checking the code (代码) (代码) of > >> dpif_netdev_flow_put, > >>>> only > >>>>> the megaflow with pmd-id equals to PMD_ID_NULL will be installed > (安装) (安装) > >>>>> into all the PMD threads. > >>>> > >>>> Agreed, I think this is the only case it could still happen. I could > not > >>>> find any other paths. > >>>> > >>>>>> //Eelco > >>>>>> > >>>>>> > >>>>>>> Eelco Chaudron 于2022年10月19日周三 18:50写道： > >>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On 10 Oct 2022, at 9:12, Eelco Chaudron wrote: > >>>>>>>> > >>>
Re: [ovs-dev] [PATCH v4 3/3] dpif-netdev: fix inconsistent processing between ukey and megaflow
Eelco Chaudron 于2022年11月24日周四 16:34写道： > > > On 24 Nov 2022, at 1:46, Peng He wrote: > > > So do we need this patch (补丁) or not?? > > > > Guessing it's quite rare in the real production environment that we have > > two datapaths at the same time > > And I am more (更多) curious that even though we have 2 datapaths, should > the port > > id be different? Is one > > port capable of being assigned to 2 datapaths at the same time > > > > Because only when a port is assigned to 2 datapaths at the same time, we > > should worry about this race > > I think we should still add this patch (补丁) , as it’s common to have a > single DPDK datapath bridge, but it could have kernel (内核) (the bridge > itself for example) and DPDK ports. In this case when the actions of the > OpenFlow rule change there could be different actions for existing rules > not yet updated by the revalidator. > > Or do I miss the point here? > > On a single DPDK datapath, the kernel ports will be processed by the DPDK datapath also (through using AF_SOCKET socket), so in this case, we only have one datapath and we will not have this race. > //Eelco > > > Eelco Chaudron 于2022年11月23日周三 23:54写道： > > > >> > >> > >> On 19 Nov 2022, at 1:46, Peng He wrote: > >> > >>> Eelco Chaudron 于2022年11月18日周五 15:38写道： > >>> > >>>> > >>>> > >>>> On 18 Nov 2022, at 2:57, Peng He wrote: > >>>> > >>>>> Since there are possible race conditions (between the kernel (内核) > (内核) > >>>> datapath and > >>>>> userspace datapath), > >>>>> I guess this patch (补丁) (补丁) is now needed again? But two datapath > is really > >>>> rare in > >>>>> the real deployment. > >>>>> So I am not sure if we should pay attention here. > >>>> > >>>> I still think we should add this, as there seem to be a decent amount > of > >>>> times people intermix a kernel (内核) (内核) interface with a DPDK one. > For > >> example, > >>>> the bridge interface, which would be up to get routing (溃败) (溃败) > information > >> for > >>>> tunnels. > >>> > >>> > >>> In this case, bridge interfaces are attached (附加) to the userspace > datapath, > >> it > >>> will be " polled (民意调查) " by the main thread, and it's pmd-id is > NON_PMD_CORE_ID. > >>> > >>> The case that race could happen is that mix using of userspace datapath > >> and > >>> kernel (内核) datapath. When the kernel datapath receives a upcall, it > will set > >>> the pmd-id to PMD_ID_NULL. Checking the code (代码) of > dpif_netdev_flow_put, > >> only > >>> the megaflow with pmd-id equals to PMD_ID_NULL will be installed (安装) > >>> into all the PMD threads. > >> > >> Agreed, I think this is the only case it could still happen. I could not > >> find any other paths. > >> > >>>> //Eelco > >>>> > >>>> > >>>>> Eelco Chaudron 于2022年10月19日周三 18:50写道： > >>>>> > >>>>>> > >>>>>> > >>>>>> On 10 Oct 2022, at 9:12, Eelco Chaudron wrote: > >>>>>> > >>>>>>> On 8 Oct 2022, at 5:27, Peng He wrote: > >>>>>>> > >>>>>>>> Hi，Eelco > >>>>>>>> > >>>>>>>> after a second thought, I think this patch (补丁) (补丁) is not needed > >> neither, > >>>>>>>> the code (代码) (代码) here is trying to find a rule which cover the > packet, > >>>>>>>> it does not mean (意味着) (意味着) the match and action of rule equals > to the > >> ones > >>>>>>>> of the ukey. > >>>>>>>> > >>>>>>>> So the code (代码) (代码) here is just a prevention, no need to make > it > >>>> consistent > >>>>>>>> with ukey. > >>>>>>>> > >>>>>>>> but the comments above are really misleading, so I sent a new > patch (补丁) > >>>> (补丁) > >>>>>> fixing > >>>>>>>> it. > >>>>>>> > >>>>>>> Ack, will wait for the v5, and review. > >>>>>> > >>>>>> As I did not see a v
Re: [ovs-dev] [PATCH v4 3/3] dpif-netdev: fix inconsistent processing between ukey and megaflow
So do we need this patch or not?? Guessing it's quite rare in the real production environment that we have two datapaths at the same time And I am more curious that even though we have 2 datapaths, should the port id be different? Is one port capable of being assigned to 2 datapaths at the same time Because only when a port is assigned to 2 datapaths at the same time, we should worry about this race Eelco Chaudron 于2022年11月23日周三 23:54写道： > > > On 19 Nov 2022, at 1:46, Peng He wrote: > > > Eelco Chaudron 于2022年11月18日周五 15:38写道： > > > >> > >> > >> On 18 Nov 2022, at 2:57, Peng He wrote: > >> > >>> Since there are possible race conditions (between the kernel (内核) > >> datapath and > >>> userspace datapath), > >>> I guess this patch (补丁) is now needed again? But two datapath is really > >> rare in > >>> the real deployment. > >>> So I am not sure if we should pay attention here. > >> > >> I still think we should add this, as there seem to be a decent amount of > >> times people intermix a kernel (内核) interface with a DPDK one. For > example, > >> the bridge interface, which would be up to get routing (溃败) information > for > >> tunnels. > > > > > > In this case, bridge interfaces are attached to the userspace datapath, > it > > will be "polled" by the main thread, and it's pmd-id is NON_PMD_CORE_ID. > > > > The case that race could happen is that mix using of userspace datapath > and > > kernel datapath. When the kernel datapath receives a upcall, it will set > > the pmd-id to PMD_ID_NULL. Checking the code of dpif_netdev_flow_put, > only > > the megaflow with pmd-id equals to PMD_ID_NULL will be installed > > into all the PMD threads. > > Agreed, I think this is the only case it could still happen. I could not > find any other paths. > > >> //Eelco > >> > >> > >>> Eelco Chaudron 于2022年10月19日周三 18:50写道： > >>> > >>>> > >>>> > >>>> On 10 Oct 2022, at 9:12, Eelco Chaudron wrote: > >>>> > >>>>> On 8 Oct 2022, at 5:27, Peng He wrote: > >>>>> > >>>>>> Hi，Eelco > >>>>>> > >>>>>> after a second thought, I think this patch (补丁) is not needed > neither, > >>>>>> the code (代码) here is trying to find a rule which cover the packet, > >>>>>> it does not mean (意味着) the match and action of rule equals to the > ones > >>>>>> of the ukey. > >>>>>> > >>>>>> So the code (代码) here is just a prevention, no need to make it > >> consistent > >>>>>> with ukey. > >>>>>> > >>>>>> but the comments above are really misleading, so I sent a new patch > >> (补丁) > >>>> fixing > >>>>>> it. > >>>>> > >>>>> Ack, will wait for the v5, and review. > >>>> > >>>> As I did not see a v5, I reviewed the v4, and assume (假设) this patch > >> (补丁) can be > >>>> ignored (忽略) . > >>>> > >>>> //Eelco > >>>> > >>>>>> Peng He 于2022年10月3日周一 20:41写道： > >>>>>> > >>>>>>> When PMDs perform upcalls, the newly generated (生成) ukey will > replace > >>>>>>> the old, however, the newly generated (生成) mageflow will be discard > >>>>>>> to reuse the old one without checking if the actions of new and > >>>>>>> old are equal. > >>>>>>> > >>>>>>> This code (代码) prevents in case someone runs dpctl/add-flow to add > >>>>>>> a dp flow with inconsistent actions with the actions of ukey, > >>>>>>> and causes more (更多) confusion (混乱) . > >>>>>>> > >>>>>>> Signed-off-by: Peng He > >>>>>>> --- > >>>>>>> lib/dpif-netdev.c | 17 - > >>>>>>> 1 file (文件) changed, 16 insertions(+), 1 deletion(-) > >>>>>>> > >>>>>>> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c > >>>>>>> index a45b46014..b316e59ef 100644 > >>>>>>> --- a/lib/dpif-netdev.c > >>>>>>> +++ b/lib/dpif-netdev.c > >>>
Re: [ovs-dev] [ovs-dev v5 1/3] ofproto-dpif-upcall: fix push_dp_ops
Eelco Chaudron 于2022年11月24日周四 00:08写道： > > > On 22 Nov 2022, at 2:44, Peng He wrote: > > > Hi, > > > > After a second thought, I think maybe keeping INCONSISTENT just for the > > modify (修改) error is a better option. > > > > With current patch (补丁) : > > 1. > > the modify (修改) error case: > > OPERATIONAL -> INCONSISTENT -> EVICTING (驱逐) -> EVICTED (驱逐) > > 2. > > the delete error case: > > EVICTING (驱逐) -> EVICTED (驱逐) > > > > Change both to INCONSISTENT: > > > > the modify (修改) error case: > > did not change. > > > > the delete error case: > > EVICTING (驱逐) -> INCONSISTENT -> EVICTED (驱逐) ? > > this will make the state machine allows both INCONSISTENT -> EVICTING > (驱逐) and > > EVICTING (驱逐) -> INCONSISTENT transitions. > > which I guess it's more (更多) confusing ... > > > > Another solution is that, drop INCONSISTENT state, if modify (修改) > fails, just > > changes to EVICTING (驱逐) . > > and let the revalidate or sweep (扫) to take (采取) care of EVICTING (驱逐) > state ukey and > > initial (初始) another dp_ops to remove it. > > > > I now prefer the second solution, what do you think? > > Yes, the second one sounds more (更多) straightforward (直截了当) , but would it > not cause issues with the statistics? If not we should probably go with > this one. The second one will also have no statistics issues. As long as the megaflow and ukey have the same lifetime, i.e. it's impossible that ukey is dead but megaflow is alive, or reverse (逆向) , there are no statistics issues. So I think the second solution (补丁) does not introduce statistics issues. will submit a v6. > > > If we run into statistics issues, which I think we will, let’s keep the > code (代码) as in v5, but add some code comments on why the paths are > different, i.e. to keep stats happy. > > > If stats are inconsistent, we should also add a test case so further > fixes/changes will not mess this up. > > > Peng He 于2022年11月22日周二 09:01写道： > > > >> > >> Eelco Chaudron 于2022年11月18日周五 15:35写道： > >> > >>> > >>> > >>> On 18 Nov 2022, at 2:53, Peng He wrote: > >>> > >>>> Eelco Chaudron 于2022年11月16日周三 18:14写道： > >>>> > >>>>> > >>>>> > >>>>> On 6 Nov 2022, at 8:12, Peng He wrote: > >>>>> > >>>>>> push_dp_ops only handles delete ops errors but ignores (忽略) (忽略) > the modify (修改) > >>>>> (修改) > >>>>>> ops results. It's better to handle all the dp operation errors in > >>>>>> a consistent way. > >>>>>> > >>>>>> We observe in the production environment that sometimes a megaflow > >>>>>> with wrong actions keep staying in datapath. The coverage command > >>> shows > >>>>>> revalidators have dumped several times, however the correct > >>>>>> actions are not set. This implies (暗示) (暗示) that the ukey's action > does not > >>>>>> equal to the meagaflow's, i.e. revalidators think the underlying > (基础) (基础) > >>>>>> megaflow's actions are correct however they are not. > >>>>>> > >>>>>> We also check the megaflow using the ofproto/trace command, and the > >>>>>> actions are not matched with the ones in the actual magaflow. By > >>>>>> performing a revalidator/purge command, the right actions are set. > >>>>>> > >>>>>> This patch (补丁) (补丁) prevents the inconsistency by considering > modify (修改) (修改) > >>>>> failure > >>>>>> in revalidators. > >>>>>> > >>>>>> To note, we cannot perform two state transitions and change > ukey_state > >>>>>> into UKEY_EVICTED directly here, because, if we do so, the > >>>>>> sweep (扫) (扫) will remove the ukey alone and leave dp flow alive. > Later, > >>> the > >>>>>> dump will retrieve (检索) (检索) the dp flow and might even recover it. > This > >>> will > >>>>>> contribute the stats of this dp flow twice. > >>>>>> > >>>>>> Signed-off-by: Peng He > >>>>> > >>>>> Hi Peng, > >>>>> > >>>>> Thanks for looking at the statistics part, see some comments inline! > >>>>> > >>>>> In a
Re: [ovs-dev] [ovs-dev v5 1/3] ofproto-dpif-upcall: fix push_dp_ops
Hi,
After a second thought, I think maybe keeping INCONSISTENT just for the
modify error is a better option.
With current patch:
1.
the modify error case:
OPERATIONAL -> INCONSISTENT -> EVICTING -> EVICTED
2.
the delete error case:
EVICTING -> EVICTED
Change both to INCONSISTENT:
the modify error case:
did not change.
the delete error case:
EVICTING -> INCONSISTENT -> EVICTED?
this will make the state machine allows both INCONSISTENT -> EVICTING and
EVICTING -> INCONSISTENT transitions.
which I guess it's more confusing ...
Another solution is that, drop INCONSISTENT state, if modify fails, just
changes to EVICTING.
and let the revalidate or sweep to take care of EVICTING state ukey and
initial another dp_ops to remove it.
I now prefer the second solution, what do you think?
Peng He 于2022年11月22日周二 09:01写道：
>
> Eelco Chaudron 于2022年11月18日周五 15:35写道：
>
>>
>>
>> On 18 Nov 2022, at 2:53, Peng He wrote:
>>
>> > Eelco Chaudron 于2022年11月16日周三 18:14写道：
>> >
>> >>
>> >>
>> >> On 6 Nov 2022, at 8:12, Peng He wrote:
>> >>
>> >>> push_dp_ops only handles delete ops errors but ignores (忽略) the modify
>> >> (修改)
>> >>> ops results. It's better to handle all the dp operation errors in
>> >>> a consistent way.
>> >>>
>> >>> We observe in the production environment that sometimes a megaflow
>> >>> with wrong actions keep staying in datapath. The coverage command
>> shows
>> >>> revalidators have dumped several times, however the correct
>> >>> actions are not set. This implies (暗示) that the ukey's action does not
>> >>> equal to the meagaflow's, i.e. revalidators think the underlying (基础)
>> >>> megaflow's actions are correct however they are not.
>> >>>
>> >>> We also check the megaflow using the ofproto/trace command, and the
>> >>> actions are not matched with the ones in the actual magaflow. By
>> >>> performing a revalidator/purge command, the right actions are set.
>> >>>
>> >>> This patch (补丁) prevents the inconsistency by considering modify (修改)
>> >> failure
>> >>> in revalidators.
>> >>>
>> >>> To note, we cannot perform two state transitions and change ukey_state
>> >>> into UKEY_EVICTED directly here, because, if we do so, the
>> >>> sweep (扫) will remove the ukey alone and leave dp flow alive. Later,
>> the
>> >>> dump will retrieve (检索) the dp flow and might even recover it. This
>> will
>> >>> contribute the stats of this dp flow twice.
>> >>>
>> >>> Signed-off-by: Peng He
>> >>
>> >> Hi Peng,
>> >>
>> >> Thanks for looking at the statistics part, see some comments inline!
>> >>
>> >> In addition, I already acked patch (补丁) 2 out of this series, but it
>> >> mentions patch x/3, but I do not see patch 3 in this series. Is this
>> >> missing? Or are there only two patches (补丁) left?
>> >
>> >
>> > there are only two patches. the third one is about the race comments,
>> which
>> > is not in this patchset.
>> > I guess I made some mistake.
>> >
>> >
>> >>
>> >>
>> >> Cheers,
>> >>
>> >> Eelco
>> >>
>> >>
>> >>> ---
>> >>> ofproto/ofproto-dpif-upcall.c | 39
>> ++-
>> >>> 1 file (文件) changed, 25 insertions(+), 14 deletions(-)
>> >>>
>> >>> diff --git a/ofproto/ofproto-dpif-upcall.c
>> >> b/ofproto/ofproto-dpif-upcall.c
>> >>> index 7ad728adf..a7970fa9b 100644
>> >>> --- a/ofproto/ofproto-dpif-upcall.c
>> >>> +++ b/ofproto/ofproto-dpif-upcall.c
>> >>> @@ -254,6 +254,7 @@ enum ukey_state {
>> >>> UKEY_CREATED = 0,
>> >>> UKEY_VISIBLE, /* Ukey is in umap, datapath flow install
>> (安装)
>> >> is queued (队列) . */
>> >>> UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is
>> installed
>> >> (安装) . */
>> >>> +UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is modified
>> >> (修改) but failed */
>> >>> UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is
>> >> queued (队列) . */
>> >>> UKEY_EVICTED, /* Ukey is in umap, datapath flow is
>&g
Re: [ovs-dev] [ovs-dev v5 1/3] ofproto-dpif-upcall: fix push_dp_ops
Eelco Chaudron 于2022年11月18日周五 15:35写道：
>
>
> On 18 Nov 2022, at 2:53, Peng He wrote:
>
> > Eelco Chaudron 于2022年11月16日周三 18:14写道：
> >
> >>
> >>
> >> On 6 Nov 2022, at 8:12, Peng He wrote:
> >>
> >>> push_dp_ops only handles delete ops errors but ignores (忽略) the modify
> >> (修改)
> >>> ops results. It's better to handle all the dp operation errors in
> >>> a consistent way.
> >>>
> >>> We observe in the production environment that sometimes a megaflow
> >>> with wrong actions keep staying in datapath. The coverage command shows
> >>> revalidators have dumped several times, however the correct
> >>> actions are not set. This implies (暗示) that the ukey's action does not
> >>> equal to the meagaflow's, i.e. revalidators think the underlying (基础)
> >>> megaflow's actions are correct however they are not.
> >>>
> >>> We also check the megaflow using the ofproto/trace command, and the
> >>> actions are not matched with the ones in the actual magaflow. By
> >>> performing a revalidator/purge command, the right actions are set.
> >>>
> >>> This patch (补丁) prevents the inconsistency by considering modify (修改)
> >> failure
> >>> in revalidators.
> >>>
> >>> To note, we cannot perform two state transitions and change ukey_state
> >>> into UKEY_EVICTED directly here, because, if we do so, the
> >>> sweep (扫) will remove the ukey alone and leave dp flow alive. Later,
> the
> >>> dump will retrieve (检索) the dp flow and might even recover it. This
> will
> >>> contribute the stats of this dp flow twice.
> >>>
> >>> Signed-off-by: Peng He
> >>
> >> Hi Peng,
> >>
> >> Thanks for looking at the statistics part, see some comments inline!
> >>
> >> In addition, I already acked patch (补丁) 2 out of this series, but it
> >> mentions patch x/3, but I do not see patch 3 in this series. Is this
> >> missing? Or are there only two patches (补丁) left?
> >
> >
> > there are only two patches. the third one is about the race comments,
> which
> > is not in this patchset.
> > I guess I made some mistake.
> >
> >
> >>
> >>
> >> Cheers,
> >>
> >> Eelco
> >>
> >>
> >>> ---
> >>> ofproto/ofproto-dpif-upcall.c | 39 ++-
> >>> 1 file (文件) changed, 25 insertions(+), 14 deletions(-)
> >>>
> >>> diff --git a/ofproto/ofproto-dpif-upcall.c
> >> b/ofproto/ofproto-dpif-upcall.c
> >>> index 7ad728adf..a7970fa9b 100644
> >>> --- a/ofproto/ofproto-dpif-upcall.c
> >>> +++ b/ofproto/ofproto-dpif-upcall.c
> >>> @@ -254,6 +254,7 @@ enum ukey_state {
> >>> UKEY_CREATED = 0,
> >>> UKEY_VISIBLE, /* Ukey is in umap, datapath flow install (安装)
> >> is queued (队列) . */
> >>> UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed
> >> (安装) . */
> >>> +UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is modified
> >> (修改) but failed */
> >>> UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is
> >> queued (队列) . */
> >>> UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted.
> */
> >>> UKEY_DELETED, /* Ukey removed from umap, ukey free is
> >> deferred (推迟) . */
> >>> @@ -1966,6 +1967,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
> >> ukey_state dst,
> >>> * UKEY_VISIBLE -> UKEY_EVICTED
> >>> * A handler attempts to install (安装) the flow, but the datapath
> >> rejects it.
> >>> * Consider that the datapath has already destroyed it.
> >>> + * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
> >>> + * A revalidator modifies (修改) the flow with error returns (返回) .
> >>> + * UKEY_INCONSISTENT -> UKEY_EVICTING
> >>> + * A revalidator decides to evict (驱逐) the datapath flow.
> >>> * UKEY_OPERATIONAL -> UKEY_EVICTING
> >>> * A revalidator decides to evict (驱逐) the datapath flow.
> >>> * UKEY_EVICTING-> UKEY_EVICTED
> >>> @@ -1974,7 +1979,8 @@ transition_ukey_at(struct udpif_key *ukey, enum
> >> ukey_state dst,
> >>> * A revalidator has removed the ukey from the umap and is
> >>
Re: [ovs-dev] [PATCH v4 3/3] dpif-netdev: fix inconsistent processing between ukey and megaflow
Eelco Chaudron 于2022年11月18日周五 15:38写道：
>
>
> On 18 Nov 2022, at 2:57, Peng He wrote:
>
> > Since there are possible race conditions (between the kernel (内核)
> datapath and
> > userspace datapath),
> > I guess this patch (补丁) is now needed again? But two datapath is really
> rare in
> > the real deployment.
> > So I am not sure if we should pay attention here.
>
> I still think we should add this, as there seem to be a decent amount of
> times people intermix a kernel (内核) interface with a DPDK one. For example,
> the bridge interface, which would be up to get routing (溃败) information for
> tunnels.
In this case, bridge interfaces are attached to the userspace datapath, it
will be "polled" by the main thread, and it's pmd-id is NON_PMD_CORE_ID.
The case that race could happen is that mix using of userspace datapath and
kernel datapath. When the kernel datapath receives a upcall, it will set
the pmd-id to PMD_ID_NULL. Checking the code of dpif_netdev_flow_put, only
the megaflow with pmd-id equals to PMD_ID_NULL will be installed
into all the PMD threads.
>
>
> //Eelco
>
>
> > Eelco Chaudron 于2022年10月19日周三 18:50写道：
> >
> >>
> >>
> >> On 10 Oct 2022, at 9:12, Eelco Chaudron wrote:
> >>
> >>> On 8 Oct 2022, at 5:27, Peng He wrote:
> >>>
> >>>> Hi，Eelco
> >>>>
> >>>> after a second thought, I think this patch (补丁) is not needed neither,
> >>>> the code (代码) here is trying to find a rule which cover the packet,
> >>>> it does not mean (意味着) the match and action of rule equals to the ones
> >>>> of the ukey.
> >>>>
> >>>> So the code (代码) here is just a prevention, no need to make it
> consistent
> >>>> with ukey.
> >>>>
> >>>> but the comments above are really misleading, so I sent a new patch
> (补丁)
> >> fixing
> >>>> it.
> >>>
> >>> Ack, will wait for the v5, and review.
> >>
> >> As I did not see a v5, I reviewed the v4, and assume (假设) this patch
> (补丁) can be
> >> ignored (忽略) .
> >>
> >> //Eelco
> >>
> >>>> Peng He 于2022年10月3日周一 20:41写道：
> >>>>
> >>>>> When PMDs perform upcalls, the newly generated (生成) ukey will replace
> >>>>> the old, however, the newly generated (生成) mageflow will be discard
> >>>>> to reuse the old one without checking if the actions of new and
> >>>>> old are equal.
> >>>>>
> >>>>> This code (代码) prevents in case someone runs dpctl/add-flow to add
> >>>>> a dp flow with inconsistent actions with the actions of ukey,
> >>>>> and causes more (更多) confusion (混乱) .
> >>>>>
> >>>>> Signed-off-by: Peng He
> >>>>> ---
> >>>>> lib/dpif-netdev.c | 17 -
> >>>>> 1 file (文件) changed, 16 insertions(+), 1 deletion(-)
> >>>>>
> >>>>> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> >>>>> index a45b46014..b316e59ef 100644
> >>>>> --- a/lib/dpif-netdev.c
> >>>>> +++ b/lib/dpif-netdev.c
> >>>>> @@ -8304,7 +8304,22 @@ handle_packet_upcall(struct
> dp_netdev_pmd_thread
> >>>>> *pmd,
> >>>>> * to be locking revalidators out of making flow
> >> modifications. */
> >>>>> ovs_mutex_lock(>flow_mutex);
> >>>>> netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> >>>>> -if (OVS_LIKELY(!netdev_flow)) {
> >>>>> +if (OVS_UNLIKELY(netdev_flow)) {
> >>>>> +struct dp_netdev_actions *old_act =
> >>>>> +dp_netdev_flow_get_actions(netdev_flow);
> >>>>> +
> >>>>> +if ((add_actions->size != old_act->size) ||
> >>>>> +memcmp(old_act->actions, add_actions->data,
> >>>>> + add_actions->size)) {
> >>>>> +
> >>>>> + struct dp_netdev_actions *new_act =
> >>>>> + dp_netdev_actions_create(add_actions->data,
> >>>>> +add_actions->size);
> >>>>> +
> >>>>> + ovsrcu_set(_flow->actions, new_act);
> >>>>> + ovsrcu_postpone(dp_netdev_actions_free, old_act);
> >>>>> +}
> >>>>> +} else {
> >>>>> netdev_flow = dp_netdev_flow_add(pmd, , ,
> >>>>> add_actions->data,
> >>>>> add_actions->size,
> >>>>> orig_in_port);
> >>>>> --
> >>>>> 2.25.1
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> hepeng
> >>
> >>
> >
> > --
> > hepeng
>
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v4 3/3] dpif-netdev: fix inconsistent processing between ukey and megaflow
Since there are possible race conditions (between the kernel datapath and
userspace datapath),
I guess this patch is now needed again? But two datapath is really rare in
the real deployment.
So I am not sure if we should pay attention here.
Eelco Chaudron 于2022年10月19日周三 18:50写道：
>
>
> On 10 Oct 2022, at 9:12, Eelco Chaudron wrote:
>
> > On 8 Oct 2022, at 5:27, Peng He wrote:
> >
> >> Hi，Eelco
> >>
> >> after a second thought, I think this patch is not needed neither,
> >> the code here is trying to find a rule which cover the packet,
> >> it does not mean the match and action of rule equals to the ones
> >> of the ukey.
> >>
> >> So the code here is just a prevention, no need to make it consistent
> >> with ukey.
> >>
> >> but the comments above are really misleading, so I sent a new patch
> fixing
> >> it.
> >
> > Ack, will wait for the v5, and review.
>
> As I did not see a v5, I reviewed the v4, and assume this patch can be
> ignored.
>
> //Eelco
>
> >> Peng He 于2022年10月3日周一 20:41写道：
> >>
> >>> When PMDs perform upcalls, the newly generated ukey will replace
> >>> the old, however, the newly generated mageflow will be discard
> >>> to reuse the old one without checking if the actions of new and
> >>> old are equal.
> >>>
> >>> This code prevents in case someone runs dpctl/add-flow to add
> >>> a dp flow with inconsistent actions with the actions of ukey,
> >>> and causes more confusion.
> >>>
> >>> Signed-off-by: Peng He
> >>> ---
> >>> lib/dpif-netdev.c | 17 -
> >>> 1 file changed, 16 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> >>> index a45b46014..b316e59ef 100644
> >>> --- a/lib/dpif-netdev.c
> >>> +++ b/lib/dpif-netdev.c
> >>> @@ -8304,7 +8304,22 @@ handle_packet_upcall(struct dp_netdev_pmd_thread
> >>> *pmd,
> >>> * to be locking revalidators out of making flow
> modifications. */
> >>> ovs_mutex_lock(>flow_mutex);
> >>> netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> >>> -if (OVS_LIKELY(!netdev_flow)) {
> >>> +if (OVS_UNLIKELY(netdev_flow)) {
> >>> +struct dp_netdev_actions *old_act =
> >>> +dp_netdev_flow_get_actions(netdev_flow);
> >>> +
> >>> +if ((add_actions->size != old_act->size) ||
> >>> +memcmp(old_act->actions, add_actions->data,
> >>> + add_actions->size)) {
> >>> +
> >>> + struct dp_netdev_actions *new_act =
> >>> + dp_netdev_actions_create(add_actions->data,
> >>> +add_actions->size);
> >>> +
> >>> + ovsrcu_set(_flow->actions, new_act);
> >>> + ovsrcu_postpone(dp_netdev_actions_free, old_act);
> >>> +}
> >>> +} else {
> >>> netdev_flow = dp_netdev_flow_add(pmd, , ,
> >>> add_actions->data,
> >>> add_actions->size,
> >>> orig_in_port);
> >>> --
> >>> 2.25.1
> >>>
> >>>
> >>
> >> --
> >> hepeng
>
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v5 1/3] ofproto-dpif-upcall: fix push_dp_ops
Eelco Chaudron 于2022年11月16日周三 18:14写道：
>
>
> On 6 Nov 2022, at 8:12, Peng He wrote:
>
> > push_dp_ops only handles delete ops errors but ignores (忽略) the modify
> (修改)
> > ops results. It's better to handle all the dp operation errors in
> > a consistent way.
> >
> > We observe in the production environment that sometimes a megaflow
> > with wrong actions keep staying in datapath. The coverage command shows
> > revalidators have dumped several times, however the correct
> > actions are not set. This implies (暗示) that the ukey's action does not
> > equal to the meagaflow's, i.e. revalidators think the underlying (基础)
> > megaflow's actions are correct however they are not.
> >
> > We also check the megaflow using the ofproto/trace command, and the
> > actions are not matched with the ones in the actual magaflow. By
> > performing a revalidator/purge command, the right actions are set.
> >
> > This patch (补丁) prevents the inconsistency by considering modify (修改)
> failure
> > in revalidators.
> >
> > To note, we cannot perform two state transitions and change ukey_state
> > into UKEY_EVICTED directly here, because, if we do so, the
> > sweep (扫) will remove the ukey alone and leave dp flow alive. Later, the
> > dump will retrieve (检索) the dp flow and might even recover it. This will
> > contribute the stats of this dp flow twice.
> >
> > Signed-off-by: Peng He
>
> Hi Peng,
>
> Thanks for looking at the statistics part, see some comments inline!
>
> In addition, I already acked patch (补丁) 2 out of this series, but it
> mentions patch x/3, but I do not see patch 3 in this series. Is this
> missing? Or are there only two patches (补丁) left?
there are only two patches. the third one is about the race comments, which
is not in this patchset.
I guess I made some mistake.
>
>
> Cheers,
>
> Eelco
>
>
> > ---
> > ofproto/ofproto-dpif-upcall.c | 39 ++-
> > 1 file (文件) changed, 25 insertions(+), 14 deletions(-)
> >
> > diff --git a/ofproto/ofproto-dpif-upcall.c
> b/ofproto/ofproto-dpif-upcall.c
> > index 7ad728adf..a7970fa9b 100644
> > --- a/ofproto/ofproto-dpif-upcall.c
> > +++ b/ofproto/ofproto-dpif-upcall.c
> > @@ -254,6 +254,7 @@ enum ukey_state {
> > UKEY_CREATED = 0,
> > UKEY_VISIBLE, /* Ukey is in umap, datapath flow install (安装)
> is queued (队列) . */
> > UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed
> (安装) . */
> > +UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is modified
> (修改) but failed */
> > UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is
> queued (队列) . */
> > UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
> > UKEY_DELETED, /* Ukey removed from umap, ukey free is
> deferred (推迟) . */
> > @@ -1966,6 +1967,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
> ukey_state dst,
> > * UKEY_VISIBLE -> UKEY_EVICTED
> > * A handler attempts to install (安装) the flow, but the datapath
> rejects it.
> > * Consider that the datapath has already destroyed it.
> > + * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
> > + * A revalidator modifies (修改) the flow with error returns (返回) .
> > + * UKEY_INCONSISTENT -> UKEY_EVICTING
> > + * A revalidator decides to evict (驱逐) the datapath flow.
> > * UKEY_OPERATIONAL -> UKEY_EVICTING
> > * A revalidator decides to evict (驱逐) the datapath flow.
> > * UKEY_EVICTING-> UKEY_EVICTED
> > @@ -1974,7 +1979,8 @@ transition_ukey_at(struct udpif_key *ukey, enum
> ukey_state dst,
> > * A revalidator has removed the ukey from the umap and is
> deleting it.
> > */
> > if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
> > - dst < UKEY_DELETED)) {
> > + dst < UKEY_DELETED) ||
> > +(ukey->state == UKEY_OPERATIONAL && dst == UKEY_EVICTING)) {
> > ukey->state = dst;
> > } else {
> > struct ds ds = DS_EMPTY_INITIALIZER;
> > @@ -2416,26 +2422,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op
> *ops, size_t n_ops)
> >
> > for (i = 0; i < n_ops; i++) {
> > struct ukey_op *op = [i];
> > -struct dpif_flow_stats *push, *stats, push_buf;
> > -
> > -stats = op->dop.flow_del.stats;
> > -push = _buf;
> > -
> > -if (o
Re: [ovs-dev] dpif-netdev: fix flow allocation size
Simon Horman 于2022年11月9日周三 02:48写道：
> On Fri, Nov 04, 2022 at 06:46:05AM +0000, Peng He wrote:
> > The tail of the struct dp_netdev_flow contains a whole netdev_flow_key
> > struct.
> >
> > We need to first minus the size of netdev_flow_key then add back
> > the real size of this netdev_flow_key.
>
> Hi Peng,
>
> It seems to me that as a result of this change the space allocated for
> struct dp_netdev_flow will be truncated just before the cr.flow.mf field,
> rather than just before the cf.flow field.
>
Yes, because the mask.len will be used for the space after cr.flow.mf.
the netdev_flow_key is defined with a maximum length of bytes that one
flow key can have:
/* Must be public as it is instantiated in subtable struct below. */
struct netdev_flow_key {
uint32_t hash; /* Hash function differs for different users. */
uint32_t len;/* Length of the following miniflow (incl. map).
*/
struct miniflow mf;
uint64_t buf[FLOW_MAX_PACKET_U64S];
};
that is to say, the "real" space of a netdev_flow_key
should be offsetof(struct netdev_flow_key, mf) + mask.len, not
sizeof(netdev_flow_key).
the dp_netdev_flow contains the whole netdev_flow_key as cr.key.
We should first minus the sizeof cr.key, then, use the
offsetof(struct netdev_flow_key, mf) + mask.len, not
sizeof(netdev_flow_key).
to recalculate the size.
> And that as a result space is allocated for the cr.flow.hash and
> cr.flow.len
> fields.
>
> But, based on the patch description, it is not clear to me why this change
> is being made. Perhaps it would be useful to add that information to the
> patch description.
>
cr.flow.
>
> > Signed-off-by: Peng He
> > ---
> > lib/dpif-netdev.c | 6 --
> > 1 file changed, 4 insertions(+), 2 deletions(-)
> >
> > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> > index 2c08a71c8..a8779a979 100644
> > --- a/lib/dpif-netdev.c
> > +++ b/lib/dpif-netdev.c
> > @@ -4072,7 +4072,8 @@ dp_netdev_flow_add(struct dp_netdev_pmd_thread
> *pmd,
> > && !FLOWMAP_HAS_FIELD(, regs));
> >
> > /* Do not allocate extra space. */
> > -flow = xmalloc(sizeof *flow - sizeof flow->cr.flow.mf + mask.len);
> > +flow = xmalloc(sizeof *flow - sizeof(flow->cr.flow) +
> > + offsetof(struct netdev_flow_key, mf) + mask.len);
> > memset(>stats, 0, sizeof flow->stats);
> > atomic_init(>netdev_flow_get_result, 0);
> > memset(>last_stats, 0, sizeof flow->last_stats);
> > @@ -9744,7 +9745,8 @@ dpcls_create_subtable(struct dpcls *cls, const
> struct netdev_flow_key *mask)
> >
> > /* Need to add one. */
> > subtable = xmalloc(sizeof *subtable
> > - - sizeof subtable->mask.mf + mask->len);
> > + - sizeof subtable->mask
> > + + offsetof(struct netdev_flow_key, mf) +
> mask->len);
> > cmap_init(>rules);
> > subtable->hit_cnt = 0;
> > netdev_flow_key_clone(>mask, mask);
> > --
> > 2.25.1
> >
> > ___
> > dev mailing list
> > d...@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] dpif-netdev: fix flow allocation size
Mike Pattrick 于2022年11月9日周三 02:09写道： > On Fri, Nov 4, 2022 at 2:46 AM Peng He wrote: > > > > The tail of the struct dp_netdev_flow contains a whole netdev_flow_key > > struct. > > > > We need to first minus the size of netdev_flow_key then add back > > the real size of this netdev_flow_key. > > > > Signed-off-by: Peng He > > --- > > lib/dpif-netdev.c | 6 -- > > 1 file changed, 4 insertions(+), 2 deletions(-) > > > > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c > > index 2c08a71c8..a8779a979 100644 > > --- a/lib/dpif-netdev.c > > +++ b/lib/dpif-netdev.c > > @@ -4072,7 +4072,8 @@ dp_netdev_flow_add(struct dp_netdev_pmd_thread > *pmd, > > && !FLOWMAP_HAS_FIELD(, regs)); > > > > /* Do not allocate extra space. */ > > -flow = xmalloc(sizeof *flow - sizeof flow->cr.flow.mf + mask.len); > > +flow = xmalloc(sizeof *flow - sizeof(flow->cr.flow) + > > I can confirm that the vlan limit test seems to be flaky, but > stylistically, it should be "sizeof flow->cr.flow" here, without the > parenthesis. > ok, I can change it to that. > > Cheers, > M > > > + offsetof(struct netdev_flow_key, mf) + mask.len); > > memset(>stats, 0, sizeof flow->stats); > > atomic_init(>netdev_flow_get_result, 0); > > memset(>last_stats, 0, sizeof flow->last_stats); > > @@ -9744,7 +9745,8 @@ dpcls_create_subtable(struct dpcls *cls, const > struct netdev_flow_key *mask) > > > > /* Need to add one. */ > > subtable = xmalloc(sizeof *subtable > > - - sizeof subtable->mask.mf + mask->len); > > + - sizeof subtable->mask > > + + offsetof(struct netdev_flow_key, mf) + > mask->len); > > cmap_init(>rules); > > subtable->hit_cnt = 0; > > netdev_flow_key_clone(>mask, mask); > > -- > > 2.25.1 > > > > ___ > > dev mailing list > > d...@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v5 2/3] ofproto-dpif-upcall: new ukey needs to take the old ukey's dump seq
The userspace datapath mananges all the magaflows by a cmap. The cmap data structrue will grow/shrink during the datapath processing and it will re-position megaflows. This might result in two revalidator threads might process a same megaflow during one dump stage. Consider a situation that, revalidator 1 processes a megaflow A, and decides to delete it from the datapath, at the mean time, this megaflow A is also queued in the process batch of revalidator 2. Normally it's ok for revalidators to process the same megaflow multiple times, as the dump_seq shows it's already dumped and the stats will not be contributed twice. Assume that right after A is deleted, a PMD thread generates again a new megaflow B which has the same match and action of A. The ukey of megaflow B will replace the one of megaflow A. Now the ukey B is new to the revalidator system and its dump seq is 0. Now since the dump seq of ukey B is 0, when processing megaflow A, the revalidator 2 will not identify this megaflow A has already been dumped by revalidator 1 and will contribute the old megaflow A's stats again, this results in an inconsistent stats between ukeys and megaflows. To fix this, the newly generated the ukey B should take the dump_seq of the replaced ukey A to avoid a same megaflow being revalidated twice in one dump stage. We observe in the production environment, the OpenFlow rules' stats sometimes are amplified compared to the actual value. Signed-off-by: Peng He Acked-by: Eelco Chaudron --- ofproto/ofproto-dpif-upcall.c | 1 + 1 file changed, 1 insertion(+) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index a7970fa9b..8089a6aa1 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -1878,6 +1878,7 @@ try_ukey_replace(struct umap *umap, struct udpif_key *old_ukey, ovs_mutex_lock(_ukey->mutex); cmap_replace(>cmap, _ukey->cmap_node, _ukey->cmap_node, new_ukey->hash); +new_ukey->dump_seq = old_ukey->dump_seq; ovsrcu_postpone(ukey_delete__, old_ukey); transition_ukey(old_ukey, UKEY_DELETED); transition_ukey(new_ukey, UKEY_VISIBLE); -- 2.25.1 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [ovs-dev v5 1/3] ofproto-dpif-upcall: fix push_dp_ops
push_dp_ops only handles delete ops errors but ignores the modify
ops results. It's better to handle all the dp operation errors in
a consistent way.
We observe in the production environment that sometimes a megaflow
with wrong actions keep staying in datapath. The coverage command shows
revalidators have dumped several times, however the correct
actions are not set. This implies that the ukey's action does not
equal to the meagaflow's, i.e. revalidators think the underlying
megaflow's actions are correct however they are not.
We also check the megaflow using the ofproto/trace command, and the
actions are not matched with the ones in the actual magaflow. By
performing a revalidator/purge command, the right actions are set.
This patch prevents the inconsistency by considering modify failure
in revalidators.
To note, we cannot perform two state transitions and change ukey_state
into UKEY_EVICTED directly here, because, if we do so, the
sweep will remove the ukey alone and leave dp flow alive. Later, the
dump will retrieve the dp flow and might even recover it. This will
contribute the stats of this dp flow twice.
Signed-off-by: Peng He
---
ofproto/ofproto-dpif-upcall.c | 39 ++-
1 file changed, 25 insertions(+), 14 deletions(-)
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index 7ad728adf..a7970fa9b 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -254,6 +254,7 @@ enum ukey_state {
UKEY_CREATED = 0,
UKEY_VISIBLE, /* Ukey is in umap, datapath flow install is queued. */
UKEY_OPERATIONAL, /* Ukey is in umap, datapath flow is installed. */
+UKEY_INCONSISTENT, /* Ukey is in umap, datapath flow is modified but
failed */
UKEY_EVICTING, /* Ukey is in umap, datapath flow delete is queued. */
UKEY_EVICTED, /* Ukey is in umap, datapath flow is deleted. */
UKEY_DELETED, /* Ukey removed from umap, ukey free is deferred. */
@@ -1966,6 +1967,10 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* UKEY_VISIBLE -> UKEY_EVICTED
* A handler attempts to install the flow, but the datapath rejects it.
* Consider that the datapath has already destroyed it.
+ * UKEY_OPERATIONAL -> UKEY_INCONSISTENT
+ * A revalidator modifies the flow with error returns.
+ * UKEY_INCONSISTENT -> UKEY_EVICTING
+ * A revalidator decides to evict the datapath flow.
* UKEY_OPERATIONAL -> UKEY_EVICTING
* A revalidator decides to evict the datapath flow.
* UKEY_EVICTING-> UKEY_EVICTED
@@ -1974,7 +1979,8 @@ transition_ukey_at(struct udpif_key *ukey, enum
ukey_state dst,
* A revalidator has removed the ukey from the umap and is deleting it.
*/
if (ukey->state == dst - 1 || (ukey->state == UKEY_VISIBLE &&
- dst < UKEY_DELETED)) {
+ dst < UKEY_DELETED) ||
+(ukey->state == UKEY_OPERATIONAL && dst == UKEY_EVICTING)) {
ukey->state = dst;
} else {
struct ds ds = DS_EMPTY_INITIALIZER;
@@ -2416,26 +2422,31 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops,
size_t n_ops)
for (i = 0; i < n_ops; i++) {
struct ukey_op *op = [i];
-struct dpif_flow_stats *push, *stats, push_buf;
-
-stats = op->dop.flow_del.stats;
-push = _buf;
-
-if (op->dop.type != DPIF_OP_FLOW_DEL) {
-/* Only deleted flows need their stats pushed. */
-continue;
-}
if (op->dop.error) {
-/* flow_del error, 'stats' is unusable. */
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
-transition_ukey(op->ukey, UKEY_EVICTED);
+if (op->dop.type == DPIF_OP_FLOW_DEL)
+transition_ukey(op->ukey, UKEY_EVICTED);
+else {
+transition_ukey(op->ukey, UKEY_INCONSISTENT);
+}
ovs_mutex_unlock(>ukey->mutex);
}
+/* if modify or delete fails, there is no need to push stats */
continue;
}
+if (op->dop.type != DPIF_OP_FLOW_DEL) {
+/* Only deleted flows need their stats pushed. */
+continue;
+}
+
+struct dpif_flow_stats *push, *stats, push_buf;
+
+stats = op->dop.flow_del.stats;
+push = _buf;
+
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
transition_ukey(op->ukey, UKEY_EVICTED);
@@ -2848,14 +2859,14 @@ revalidator_sweep__(struct revalidator *revalidator,
bool purge)
continue;
}
ukey_state = ukey->state;
-if (ukey_state == UKEY_OPERATIONAL
+if (ukey_state == UKEY_OPERATIONAL || ukey_state ==
UKEY_
Re: [ovs-dev] dpif-netdev: fix flow allocation size
Hi, I have re-runed the test myself, and it passed. Guess this test is a flaky test. Peng He 于2022年11月4日周五 14:46写道： > The tail of the struct dp_netdev_flow contains a whole netdev_flow_key > struct. > > We need to first minus the size of netdev_flow_key then add back > the real size of this netdev_flow_key. > > Signed-off-by: Peng He > --- > lib/dpif-netdev.c | 6 -- > 1 file changed, 4 insertions(+), 2 deletions(-) > > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c > index 2c08a71c8..a8779a979 100644 > --- a/lib/dpif-netdev.c > +++ b/lib/dpif-netdev.c > @@ -4072,7 +4072,8 @@ dp_netdev_flow_add(struct dp_netdev_pmd_thread *pmd, > && !FLOWMAP_HAS_FIELD(, regs)); > > /* Do not allocate extra space. */ > -flow = xmalloc(sizeof *flow - sizeof flow->cr.flow.mf + mask.len); > +flow = xmalloc(sizeof *flow - sizeof(flow->cr.flow) + > + offsetof(struct netdev_flow_key, mf) + mask.len); > memset(>stats, 0, sizeof flow->stats); > atomic_init(>netdev_flow_get_result, 0); > memset(>last_stats, 0, sizeof flow->last_stats); > @@ -9744,7 +9745,8 @@ dpcls_create_subtable(struct dpcls *cls, const > struct netdev_flow_key *mask) > > /* Need to add one. */ > subtable = xmalloc(sizeof *subtable > - - sizeof subtable->mask.mf + mask->len); > + - sizeof subtable->mask > + + offsetof(struct netdev_flow_key, mf) + > mask->len); > cmap_init(>rules); > subtable->hit_cnt = 0; > netdev_flow_key_clone(>mask, mask); > -- > 2.25.1 > > -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-build] |fail| pw1699340 dpif-netdev: fix flow allocation size
I've rerun the test 150 in my VM. it runs successfully. Looks like this is a flaky uint test. Peng He 于2022年11月4日周五 17:05写道： > I've rerun the test 150 in my VM. it runs successfully. > Looks like this is a flaky uint test. > > 于2022年11月4日周五 16:24写道： > >> Test-Label: intel-ovs-compilation >> Test-Status: fail >> http://patchwork.ozlabs.org/api/patches/1699340/ >> >> AVX-512_compilation: failed >> DPLCS Test: fail >> DPIF Test: success >> MFEX Test: success >> Errors in DPCLS test: >> make check-system-userspace TESTSUITEFLAGS='1-148 150-160' >> make all-am >> make[1]: Entering directory '/root/ovs-dev' >> make[1]: Leaving directory '/root/ovs-dev' >> set /bin/bash './tests/system-userspace-testsuite' -C tests >> AUTOTEST_PATH='utilities:vswitchd:ovsdb:vtep:tests:ipsec::'; \ >> "$@" 1-148 150-160 -j1 || (test X'' = Xyes && "$@" --recheck) >> Illegal "police" >> ## -- ## >> ## openvswitch 3.0.90 test suite. ## >> ## -- ## >> >> datapath-sanity >> >> 1: datapath - ping between two ports ok >> 2: datapath - http between two ports ok >> 3: datapath - ping between two ports on vlan ok >> 4: datapath - ping between two ports on cvlan skipped ( >> system-traffic.at:74) >> 5: datapath - ping6 between two ports ok >> 6: datapath - ping6 between two ports on vlan ok >> 7: datapath - ping6 between two ports on cvlan skipped ( >> system-traffic.at:165) >> 8: datapath - ping6 between two ports IPv6 later fragments ok >> 9: datapath - ping6 between two ports with header modify ok >> 10: datapath - ping over bond ok >> 11: datapath - ping over vxlan tunnel ok >> 12: datapath - ping vlan over vxlan tunnel ok >> 13: datapath - ping over vxlan6 tunnel ok >> 14: datapath - ping over gre tunnel skipped ( >> system-traffic.at:429) >> 15: datapath - ping over ip6gre L2 tunnel skipped ( >> system-traffic.at:471) >> 16: datapath - ping over erspan v1 tunnel skipped ( >> system-traffic.at:512) >> 17: datapath - ping over erspan v2 tunnel skipped ( >> system-traffic.at:549) >> 18: datapath - ping over ip6erspan v1 tunnelskipped ( >> system-traffic.at:586) >> 19: datapath - ping over ip6erspan v2 tunnelskipped ( >> system-traffic.at:626) >> 20: datapath - ping over geneve tunnel ok >> 21: datapath - ping over geneve tunnel, delete flow regression ok >> 22: datapath - flow resume with geneve tun_metadata ok >> 23: datapath - ping over geneve6 tunnel ok >> 24: datapath - ping over gre tunnel by simulated packets skipped ( >> system-traffic.at:860) >> 25: datapath - ping over erspan v1 tunnel by simulated packets skipped ( >> system-traffic.at:907) >> 26: datapath - ping over erspan v2 tunnel by simulated packets skipped ( >> system-traffic.at:956) >> 27: datapath - ping over ip6erspan v1 tunnel by simulated packets >> skipped (system-traffic.at:1010) >> 28: datapath - ping over ip6erspan v2 tunnel by simulated packets >> skipped (system-traffic.at:1066) >> 29: datapath - clone action ok >> 30: datapath - mpls actions ok >> 31: datapath - multiple mpls label pop ok >> 32: datapath - encap decap mpls actions ok >> 33: datapath - encap decap mpls_mc actions ok >> 34: datapath - multiple encap decap mpls actionsok >> 35: datapath - multiple encap decap mpls_mc actions ok >> 36: datapath - encap mpls pop mpls actions ok >> 37: datapath - push mpls decap mpls actions ok >> 38: datapath - basic truncate actionok >> 39: datapath - truncate and output to gre tunnel by simulated packets >> skipped (system-traffic.at:1592) >> 40: datapath - truncate and output to gre tunnelskipped ( >> system-traffic.at:1725) >> 41: datapath - configure cache size skipped ( >> system-traffic.at:1846) >> >> MPLS >> >> 42: mpls - encap header dp-support ok >> 43: mpls - encap header slow-path ok >> 44: mpls_mc - encap header dp-support ok >> 45: mpls_mc - encap header slow-pathok >> 46: mpls - decap header dp-support ok >> 47: mpls - decap header slow-p
[ovs-dev] dpif-netdev: fix flow allocation size
The tail of the struct dp_netdev_flow contains a whole netdev_flow_key struct. We need to first minus the size of netdev_flow_key then add back the real size of this netdev_flow_key. Signed-off-by: Peng He --- lib/dpif-netdev.c | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 2c08a71c8..a8779a979 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -4072,7 +4072,8 @@ dp_netdev_flow_add(struct dp_netdev_pmd_thread *pmd, && !FLOWMAP_HAS_FIELD(, regs)); /* Do not allocate extra space. */ -flow = xmalloc(sizeof *flow - sizeof flow->cr.flow.mf + mask.len); +flow = xmalloc(sizeof *flow - sizeof(flow->cr.flow) + + offsetof(struct netdev_flow_key, mf) + mask.len); memset(>stats, 0, sizeof flow->stats); atomic_init(>netdev_flow_get_result, 0); memset(>last_stats, 0, sizeof flow->last_stats); @@ -9744,7 +9745,8 @@ dpcls_create_subtable(struct dpcls *cls, const struct netdev_flow_key *mask) /* Need to add one. */ subtable = xmalloc(sizeof *subtable - - sizeof subtable->mask.mf + mask->len); + - sizeof subtable->mask + + offsetof(struct netdev_flow_key, mf) + mask->len); cmap_init(>rules); subtable->hit_cnt = 0; netdev_flow_key_clone(>mask, mask); -- 2.25.1 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v4 1/3] ofproto-dpif-upcall: fix push_dp_ops
sorry to miss that part,
will take some time to think if we need it or not.
BTW, looks like the kernel handle_upcalls code path has also this issue.
the code is here:
for (i = 0; i < n_ops; i++) {
struct udpif_key *ukey = ops[i].ukey;
if (ukey) {
ovs_mutex_lock(>mutex);
if (ops[i].dop.error) {
transition_ukey(ukey, UKEY_EVICTED);
} else if (ukey->state < UKEY_OPERATIONAL) {
transition_ukey(ukey, UKEY_OPERATIONAL);
}
ovs_mutex_unlock(>mutex);
}
}
Eelco Chaudron 于2022年10月19日周三 18:44写道：
>
>
> On 3 Oct 2022, at 6:02, Peng He wrote:
>
> > push_dp_ops only handles delete ops errors but ignores the modify
> > ops results. It's better to handle all the dp operation errors in
> > a consistent way.
> >
> > We observe in the production environment that sometimes a megaflow
> > with wrong actions keep staying in datapath. The coverage command shows
> > revalidators have dumped several times, however the correct
> > actions are not set. This implies that the ukey's action does not
> > equal to the meagaflow's, i.e. revalidators think the underlying
> > megaflow's actions are correct however they are not.
> >
> > We also check the megaflow using the ofproto/trace command, and the
> > actions are not matched with the ones in the actual magaflow. By
> > performing a revalidator/purge command, the right actions are set.
> >
> > This patch prevents the inconsistency by considering modify failure
> > in revalidators.
> >
> > Signed-off-by: Peng He
> > ---
> > ofproto/ofproto-dpif-upcall.c | 37 ---
> > 1 file changed, 26 insertions(+), 11 deletions(-)
> >
> > diff --git a/ofproto/ofproto-dpif-upcall.c
> b/ofproto/ofproto-dpif-upcall.c
> > index 7ad728adf..dd1abfdee 100644
> > --- a/ofproto/ofproto-dpif-upcall.c
> > +++ b/ofproto/ofproto-dpif-upcall.c
> > @@ -2416,26 +2416,41 @@ push_dp_ops(struct udpif *udpif, struct ukey_op
> *ops, size_t n_ops)
> >
> > for (i = 0; i < n_ops; i++) {
> > struct ukey_op *op = [i];
> > -struct dpif_flow_stats *push, *stats, push_buf;
> > -
> > -stats = op->dop.flow_del.stats;
> > -push = _buf;
> > -
> > -if (op->dop.type != DPIF_OP_FLOW_DEL) {
> > -/* Only deleted flows need their stats pushed. */
> > -continue;
> > -}
> >
> > if (op->dop.error) {
> > -/* flow_del error, 'stats' is unusable. */
> > if (op->ukey) {
> > ovs_mutex_lock(>ukey->mutex);
> > -transition_ukey(op->ukey, UKEY_EVICTED);
> > +if (op->dop.type == DPIF_OP_FLOW_DEL)
> > +transition_ukey(op->ukey, UKEY_EVICTED);
> > +else {
> > +/* modify error has variety reasons, it's easy
> > + * to transition the state into UKEY_EVICTED, and
> > + * let revalidator_sweep remove the ukey.
> > + *
> > + * In the next round of revalidating, the
> revalidator
> > + * can recover the ukey through
> ukey_create_from_dpif_flow,
> > + * if recover fails, the revalidator will remove
> the datapath
> > + * flow itself.
> > + */
> > +transition_ukey(op->ukey, UKEY_EVICTING);
> > +transition_ukey(op->ukey, UKEY_EVICTED);
> > +}
> > ovs_mutex_unlock(>ukey->mutex);
> > }
> > +/* if modify or delete fails, there is no need to push
> stats */
> > continue;
> > }
> >
> > +if (op->dop.type != DPIF_OP_FLOW_DEL) {
> > +/* Only deleted flows need their stats pushed. */
>
> The code looks good, however, what was the conclusion on updating stats on
> other flows that now got deleted instead of updated? Do they not need their
> stats pushed? See v3 discussion.
>
> > +continue;
> > +}
> > +
> > +struct dpif_flow_stats *push, *stats, push_buf;
> > +
> > +stats = op->dop.flow_del.stats;
> > +push = _buf;
> > +
> > if (op->ukey) {
> > ovs_mutex_lock(>ukey->mutex);
> > transition_ukey(op->ukey, UKEY_EVICTED);
> > --
> > 2.25.1
>
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH] dpif-netdev: fix the race comments
I'll take a look at that!
thanks.
Eelco Chaudron 于2022年10月19日周三 18:21写道：
>
>
> On 3 Oct 2022, at 6:02, Peng He wrote:
>
> > The following comments (brought in at 0de8783a9):
> >
> > /* XXX: There's a race window where a flow covering this packet
> > * could have already been installed since we last did the flow
> > * lookup before upcall. This could be solved by moving the
> > * mutex lock outside the loop, but that's an awful long time
> > * to be locking revalidators out of making flow modifications. */
> >
> > is out-dated. Back at commit 0de8783a9, the classifier is per-datapath,
> > multiple PMDs share a same classifier. Since now we have changed into
> > per-PMD classifier, the lookup code only prevents from the race
> > introduced by manually calling dpctl/add-flow, there are no other
> > threads which would insert datapath flows.
> >
> > Signed-off-by: Peng He
> > ---
> > lib/dpif-netdev.c | 8 +++-
> > 1 file changed, 3 insertions(+), 5 deletions(-)
> >
> > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> > index b316e59ef..680321600 100644
> > --- a/lib/dpif-netdev.c
> > +++ b/lib/dpif-netdev.c
> > @@ -8297,12 +8297,10 @@ handle_packet_upcall(struct dp_netdev_pmd_thread
> *pmd,
> > if (OVS_LIKELY(error != ENOSPC)) {
> > struct dp_netdev_flow *netdev_flow;
> >
> > -/* XXX: There's a race window where a flow covering this packet
> > - * could have already been installed since we last did the flow
> > - * lookup before upcall. This could be solved by moving the
> > - * mutex lock outside the loop, but that's an awful long time
> > - * to be locking revalidators out of making flow modifications.
> */
> > ovs_mutex_lock(>flow_mutex);
> > +/* We need to lookup the flow in case someone uses
> dpctl/add-flow
> > + * to add a datapath flow and this flow also cover this packet.
> > + */
>
> I actually was looking for code paths that could hit this case, but was
> not able to quickly identify anything but this one. However are we sure
> there is no such odd kernel tunnel-like interface that could take this
> path? However, if this happens it’s still being done in the same thread
> that handles the dpctl/add-flow flow.
>
> See the discussion in
> https://mail.openvswitch.org/pipermail/ovs-dev/2022-October/398619.html.
>
> //Eelco
>
> > netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > if (OVS_UNLIKELY(netdev_flow)) {
> > struct dp_netdev_actions *old_act =
> > --
> > 2.25.1
>
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v3 3/4] ofproto-dpif-upcall: new ukey needs to take the old ukey's dump seq
Hi，Eelco,
Which kernel version you have tested on?
I've also tested the #50 for several hundreds times, but only get 0 packets
once.
I have put your patches upon on commits 49efc63ad: ofproto-dpif-xlate: Fix
error messages for nonexistent ports/recirc_ids.
not sure if it's related to the modification on master branch.
Eelco Chaudron 于2022年10月7日周五 18:54写道：
> Hi hepeng,
>
> I tried your changes, but I also do get a lot of zero packet failures on
> test 50. Not sure what happened, maybe it was related to syncing to the
> latest master.
>
> However I focused on test 49, and I can replicate it every x test, but
> within 40 tries:
>
> ## -- ##
> ## openvswitch 3.0.90 test suite. ##
> ## -- ##
> 49: datapath - truncate and output to gre tunnel by simulated packets
> FAILED (ovs-macros.at:247)
>
> ...
> ...
> system-traffic.at:1663: waiting until ovs-ofctl dump-flows br-underlay |
> grep "in_port=LOCAL" | sed -n 's/.*\(n\_bytes=[0-9]*\).*/\1/p' | grep
> "n_bytes=138"...
> system-traffic.at:1663: wait failed after 30 seconds
> n_bytes=18446744073707612670
> ./ovs-macros.at:247: hard failure
>
> I’ll try to dig into this later once I get some time...
>
> //Eelco
>
> On 2 Oct 2022, at 8:40, Peng He wrote:
>
> Hi,
>
> I have tried test #50 for 200 times, the overflow as well as stats of 0
> did not show up.
>
> the command line is:
> for i in {1..100}; do echo $i; make check-offloads TESTSUITEFLAGS="-v 50"
> || break; done
>
> I am using Debian 10, and my kernel version is
> root@ubuntu:~/ovs# uname -r
> 5.4.0-121-generic
>
> I doubt the overflow might be related to kernel datapath.
> IIRC, the kernel datapath does not support some IPv6 ND/NA match.
>
> In the logs, there are some warning saying that "failed to get/del/put
> some datapath flow", which is all
> related the ICMPv6. It seems to prove the missing of the support of NA/ND
> match in datapath.
>
> since the ICMPv6 is not related to the test itself. I did some
> modification to the test to drop all IPv6 traffic.
>
> Below is my complete diff to the #50:
>
> diff --git a/tests/system-traffic.at b/tests/system-traffic.at
> index dfbc30e47..b759c4bb9 100644
> --- a/tests/system-traffic.at
> +++ b/tests/system-traffic.at
> @@ -1762,7 +1762,7 @@ on_exit 'rm -f payload200.bin'
>
> AT_CHECK([ovs-ofctl del-flows br0])
> AT_DATA([flows.txt], [dnl
>
> -priority=99,in_port=1,actions=output(port=2,max_len=100),output(port=3,max_len=100)
>
> +priority=99,in_port=1,ip,actions=output(port=2,max_len=100),output(port=3,max_len=100)
> priority=99,in_port=2,udp,actions=output(port=1,max_len=100)
> priority=1,in_port=4,ip,actions=drop
> priority=1,actions=drop
> @@ -1771,6 +1771,8 @@ AT_CHECK([ovs-ofctl add-flows br0 flows.txt])
>
> AT_CHECK([ovs-ofctl del-flows br-underlay])
> AT_DATA([flows-underlay.txt], [dnl
> +priority=99,arp,in_port=1,actions=LOCAL
> +priority=99,arp,in_port=LOCAL,actions=1
> priority=99,dl_type=0x0800,nw_proto=47,in_port=1,actions=LOCAL
> priority=99,dl_type=0x0800,nw_proto=47,in_port=LOCAL,ip_dst=
> 172.31.1.1/24,actions=1
> priority=1,actions=drop
> @@ -1787,7 +1789,7 @@ AT_CHECK([ovs-ofctl dump-flows br0 | grep
> "in_port=2" | sed -n 's/.*\(n\_bytes=[
> n_bytes=242
> ])
> dnl After truncation = outer ETH(14) + outer IP(20) + GRE(4) + 100 = 138B
> -AT_CHECK([ovs-ofctl dump-flows br-underlay | grep "in_port=LOCAL" | sed
> -n 's/.*\(n\_bytes=[[0-9]]*\).*/\1/p'], [0], [dnl
> +AT_CHECK([ovs-ofctl dump-flows br-underlay | grep "ip,in_port=LOCAL" |
> sed -n 's/.*\(n\_bytes=[[0-9]]*\).*/\1/p'], [0], [dnl
> n_bytes=138
> ])
>
> @@ -1822,7 +1824,7 @@ AT_CHECK([ovs-ofctl dump-flows br0 | grep
> "in_port=2" | sed -n 's/.*\(n\_bytes=[
> n_bytes=242
> ])
> dnl After truncation = outer ETH(14) + outer IP(20) + GRE(4) + 100 = 138B
> -AT_CHECK([ovs-ofctl dump-flows br-underlay | grep "in_port=LOCAL" | sed
> -n 's/.*\(n\_bytes=[[0-9]]*\).*/\1/p'], [0], [dnl
> +AT_CHECK([ovs-ofctl dump-flows br-underlay | grep "ip,in_port=LOCAL" |
> sed -n 's/.*\(n\_bytes=[[0-9]]*\).*/\1/p'], [0], [dnl
> n_bytes=138
> ])
>
> @@ -1834,7 +1836,11 @@ AT_CHECK([ovs-ofctl dump-flows br0 | grep
> "in_port=4" | ofctl_strip], [0], [dnl
> n_packets=1, n_bytes=100, priority=1,ip,in_port=4 actions=drop
> ])
>
> -OVS_TRAFFIC_VSWITCHD_STOP
> +OVS_TRAFFIC_VSWITCHD_STOP(["dnl
> +/.*lost packet on port.*/d
> +/.failed to flow_get.*/d
> +/.failed to flow_del.*/d
> +/.*Failed to acquire udpif_key corresponding to unexpected flow.*/d"])
> AT_CLEANUP
>
Re: [ovs-dev] [PATCH v4 3/3] dpif-netdev: fix inconsistent processing between ukey and megaflow
Hi，Eelco
after a second thought, I think this patch is not needed neither,
the code here is trying to find a rule which cover the packet,
it does not mean the match and action of rule equals to the ones
of the ukey.
So the code here is just a prevention, no need to make it consistent
with ukey.
but the comments above are really misleading, so I sent a new patch fixing
it.
Peng He 于2022年10月3日周一 20:41写道：
> When PMDs perform upcalls, the newly generated ukey will replace
> the old, however, the newly generated mageflow will be discard
> to reuse the old one without checking if the actions of new and
> old are equal.
>
> This code prevents in case someone runs dpctl/add-flow to add
> a dp flow with inconsistent actions with the actions of ukey,
> and causes more confusion.
>
> Signed-off-by: Peng He
> ---
> lib/dpif-netdev.c | 17 -
> 1 file changed, 16 insertions(+), 1 deletion(-)
>
> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> index a45b46014..b316e59ef 100644
> --- a/lib/dpif-netdev.c
> +++ b/lib/dpif-netdev.c
> @@ -8304,7 +8304,22 @@ handle_packet_upcall(struct dp_netdev_pmd_thread
> *pmd,
> * to be locking revalidators out of making flow modifications. */
> ovs_mutex_lock(>flow_mutex);
> netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> -if (OVS_LIKELY(!netdev_flow)) {
> +if (OVS_UNLIKELY(netdev_flow)) {
> +struct dp_netdev_actions *old_act =
> +dp_netdev_flow_get_actions(netdev_flow);
> +
> +if ((add_actions->size != old_act->size) ||
> +memcmp(old_act->actions, add_actions->data,
> + add_actions->size)) {
> +
> + struct dp_netdev_actions *new_act =
> + dp_netdev_actions_create(add_actions->data,
> +add_actions->size);
> +
> + ovsrcu_set(_flow->actions, new_act);
> + ovsrcu_postpone(dp_netdev_actions_free, old_act);
> +}
> +} else {
> netdev_flow = dp_netdev_flow_add(pmd, , ,
> add_actions->data,
> add_actions->size,
> orig_in_port);
> --
> 2.25.1
>
>
--
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [PATCH v4 3/3] dpif-netdev: fix inconsistent processing between ukey and megaflow
When PMDs perform upcalls, the newly generated ukey will replace
the old, however, the newly generated mageflow will be discard
to reuse the old one without checking if the actions of new and
old are equal.
This code prevents in case someone runs dpctl/add-flow to add
a dp flow with inconsistent actions with the actions of ukey,
and causes more confusion.
Signed-off-by: Peng He
---
lib/dpif-netdev.c | 17 -
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index a45b46014..b316e59ef 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -8304,7 +8304,22 @@ handle_packet_upcall(struct dp_netdev_pmd_thread *pmd,
* to be locking revalidators out of making flow modifications. */
ovs_mutex_lock(>flow_mutex);
netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
-if (OVS_LIKELY(!netdev_flow)) {
+if (OVS_UNLIKELY(netdev_flow)) {
+struct dp_netdev_actions *old_act =
+dp_netdev_flow_get_actions(netdev_flow);
+
+if ((add_actions->size != old_act->size) ||
+memcmp(old_act->actions, add_actions->data,
+ add_actions->size)) {
+
+ struct dp_netdev_actions *new_act =
+ dp_netdev_actions_create(add_actions->data,
+add_actions->size);
+
+ ovsrcu_set(_flow->actions, new_act);
+ ovsrcu_postpone(dp_netdev_actions_free, old_act);
+}
+} else {
netdev_flow = dp_netdev_flow_add(pmd, , ,
add_actions->data,
add_actions->size, orig_in_port);
--
2.25.1
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [PATCH v4 2/3] ofproto-dpif-upcall: new ukey needs to take the old ukey's dump seq
The userspace datapath mananges all the magaflows by a cmap. The cmap data structrue will grow/shrink during the datapath processing and it will re-position megaflows. This might result in two revalidator threads might process a same megaflow during one dump stage. Consider a situation that, revalidator 1 processes a megaflow A, and decides to delete it from the datapath, at the mean time, this megaflow A is also queued in the process batch of revalidator 2. Normally it's ok for revalidators to process the same megaflow multiple times, as the dump_seq shows it's already dumped and the stats will not be contributed twice. Assume that right after A is deleted, a PMD thread generates again a new megaflow B which has the same match and action of A. The ukey of megaflow B will replace the one of megaflow A. Now the ukey B is new to the revalidator system and its dump seq is 0. Now since the dump seq of ukey B is 0, when processing megaflow A, the revalidator 2 will not identify this megaflow A has already been dumped by revalidator 1 and will contribute the old megaflow A's stats again, this results in an inconsistent stats between ukeys and megaflows. To fix this, the newly generated the ukey B should take the dump_seq of the replaced ukey A to avoid a same megaflow being revalidated twice in one dump stage. We observe in the production environment, the OpenFlow rules' stats sometimes are amplified compared to the actual value. Signed-off-by: Peng He --- ofproto/ofproto-dpif-upcall.c | 1 + 1 file changed, 1 insertion(+) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index dd1abfdee..e1af4d633 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -1877,6 +1877,7 @@ try_ukey_replace(struct umap *umap, struct udpif_key *old_ukey, ovs_mutex_lock(_ukey->mutex); cmap_replace(>cmap, _ukey->cmap_node, _ukey->cmap_node, new_ukey->hash); +new_ukey->dump_seq = old_ukey->dump_seq; ovsrcu_postpone(ukey_delete__, old_ukey); transition_ukey(old_ukey, UKEY_DELETED); transition_ukey(new_ukey, UKEY_VISIBLE); -- 2.25.1 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [PATCH v4 1/3] ofproto-dpif-upcall: fix push_dp_ops
push_dp_ops only handles delete ops errors but ignores the modify
ops results. It's better to handle all the dp operation errors in
a consistent way.
We observe in the production environment that sometimes a megaflow
with wrong actions keep staying in datapath. The coverage command shows
revalidators have dumped several times, however the correct
actions are not set. This implies that the ukey's action does not
equal to the meagaflow's, i.e. revalidators think the underlying
megaflow's actions are correct however they are not.
We also check the megaflow using the ofproto/trace command, and the
actions are not matched with the ones in the actual magaflow. By
performing a revalidator/purge command, the right actions are set.
This patch prevents the inconsistency by considering modify failure
in revalidators.
Signed-off-by: Peng He
---
ofproto/ofproto-dpif-upcall.c | 37 ---
1 file changed, 26 insertions(+), 11 deletions(-)
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index 7ad728adf..dd1abfdee 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -2416,26 +2416,41 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops,
size_t n_ops)
for (i = 0; i < n_ops; i++) {
struct ukey_op *op = [i];
-struct dpif_flow_stats *push, *stats, push_buf;
-
-stats = op->dop.flow_del.stats;
-push = _buf;
-
-if (op->dop.type != DPIF_OP_FLOW_DEL) {
-/* Only deleted flows need their stats pushed. */
-continue;
-}
if (op->dop.error) {
-/* flow_del error, 'stats' is unusable. */
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
-transition_ukey(op->ukey, UKEY_EVICTED);
+if (op->dop.type == DPIF_OP_FLOW_DEL)
+transition_ukey(op->ukey, UKEY_EVICTED);
+else {
+/* modify error has variety reasons, it's easy
+ * to transition the state into UKEY_EVICTED, and
+ * let revalidator_sweep remove the ukey.
+ *
+ * In the next round of revalidating, the revalidator
+ * can recover the ukey through ukey_create_from_dpif_flow,
+ * if recover fails, the revalidator will remove the
datapath
+ * flow itself.
+ */
+transition_ukey(op->ukey, UKEY_EVICTING);
+transition_ukey(op->ukey, UKEY_EVICTED);
+}
ovs_mutex_unlock(>ukey->mutex);
}
+/* if modify or delete fails, there is no need to push stats */
continue;
}
+if (op->dop.type != DPIF_OP_FLOW_DEL) {
+/* Only deleted flows need their stats pushed. */
+continue;
+}
+
+struct dpif_flow_stats *push, *stats, push_buf;
+
+stats = op->dop.flow_del.stats;
+push = _buf;
+
if (op->ukey) {
ovs_mutex_lock(>ukey->mutex);
transition_ukey(op->ukey, UKEY_EVICTED);
--
2.25.1
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] [PATCH] dpif-netdev: fix the race comments
The following comments (brought in at 0de8783a9):
/* XXX: There's a race window where a flow covering this packet
* could have already been installed since we last did the flow
* lookup before upcall. This could be solved by moving the
* mutex lock outside the loop, but that's an awful long time
* to be locking revalidators out of making flow modifications. */
is out-dated. Back at commit 0de8783a9, the classifier is per-datapath,
multiple PMDs share a same classifier. Since now we have changed into
per-PMD classifier, the lookup code only prevents from the race
introduced by manually calling dpctl/add-flow, there are no other
threads which would insert datapath flows.
Signed-off-by: Peng He
---
lib/dpif-netdev.c | 8 +++-
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index b316e59ef..680321600 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -8297,12 +8297,10 @@ handle_packet_upcall(struct dp_netdev_pmd_thread *pmd,
if (OVS_LIKELY(error != ENOSPC)) {
struct dp_netdev_flow *netdev_flow;
-/* XXX: There's a race window where a flow covering this packet
- * could have already been installed since we last did the flow
- * lookup before upcall. This could be solved by moving the
- * mutex lock outside the loop, but that's an awful long time
- * to be locking revalidators out of making flow modifications. */
ovs_mutex_lock(>flow_mutex);
+/* We need to lookup the flow in case someone uses dpctl/add-flow
+ * to add a datapath flow and this flow also cover this packet.
+ */
netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
if (OVS_UNLIKELY(netdev_flow)) {
struct dp_netdev_actions *old_act =
--
2.25.1
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v3 3/4] ofproto-dpif-upcall: new ukey needs to take the old ukey's dump seq
Hi,
I have tried test #50 for 200 times, the overflow as well as stats of 0 did
not show up.
the command line is:
for i in {1..100}; do echo $i; make check-offloads TESTSUITEFLAGS="-v 50"
|| break; done
I am using Debian 10, and my kernel version is
root@ubuntu:~/ovs# uname -r
5.4.0-121-generic
I doubt the overflow might be related to kernel datapath.
IIRC, the kernel datapath does not support some IPv6 ND/NA match.
In the logs, there are some warning saying that "failed to get/del/put some
datapath flow", which is all
related the ICMPv6. It seems to prove the missing of the support of NA/ND
match in datapath.
since the ICMPv6 is not related to the test itself. I did some modification
to the test to drop all IPv6 traffic.
Below is my complete diff to the #50:
diff --git a/tests/system-traffic.at b/tests/system-traffic.at
index dfbc30e47..b759c4bb9 100644
--- a/tests/system-traffic.at
+++ b/tests/system-traffic.at
@@ -1762,7 +1762,7 @@ on_exit 'rm -f payload200.bin'
AT_CHECK([ovs-ofctl del-flows br0])
AT_DATA([flows.txt], [dnl
-priority=99,in_port=1,actions=output(port=2,max_len=100),output(port=3,max_len=100)
+priority=99,in_port=1,ip,actions=output(port=2,max_len=100),output(port=3,max_len=100)
priority=99,in_port=2,udp,actions=output(port=1,max_len=100)
priority=1,in_port=4,ip,actions=drop
priority=1,actions=drop
@@ -1771,6 +1771,8 @@ AT_CHECK([ovs-ofctl add-flows br0 flows.txt])
AT_CHECK([ovs-ofctl del-flows br-underlay])
AT_DATA([flows-underlay.txt], [dnl
+priority=99,arp,in_port=1,actions=LOCAL
+priority=99,arp,in_port=LOCAL,actions=1
priority=99,dl_type=0x0800,nw_proto=47,in_port=1,actions=LOCAL
priority=99,dl_type=0x0800,nw_proto=47,in_port=LOCAL,ip_dst=
172.31.1.1/24,actions=1
priority=1,actions=drop
@@ -1787,7 +1789,7 @@ AT_CHECK([ovs-ofctl dump-flows br0 | grep "in_port=2"
| sed -n 's/.*\(n\_bytes=[
n_bytes=242
])
dnl After truncation = outer ETH(14) + outer IP(20) + GRE(4) + 100 = 138B
-AT_CHECK([ovs-ofctl dump-flows br-underlay | grep "in_port=LOCAL" | sed -n
's/.*\(n\_bytes=[[0-9]]*\).*/\1/p'], [0], [dnl
+AT_CHECK([ovs-ofctl dump-flows br-underlay | grep "ip,in_port=LOCAL" | sed
-n 's/.*\(n\_bytes=[[0-9]]*\).*/\1/p'], [0], [dnl
n_bytes=138
])
@@ -1822,7 +1824,7 @@ AT_CHECK([ovs-ofctl dump-flows br0 | grep "in_port=2"
| sed -n 's/.*\(n\_bytes=[
n_bytes=242
])
dnl After truncation = outer ETH(14) + outer IP(20) + GRE(4) + 100 = 138B
-AT_CHECK([ovs-ofctl dump-flows br-underlay | grep "in_port=LOCAL" | sed -n
's/.*\(n\_bytes=[[0-9]]*\).*/\1/p'], [0], [dnl
+AT_CHECK([ovs-ofctl dump-flows br-underlay | grep "ip,in_port=LOCAL" | sed
-n 's/.*\(n\_bytes=[[0-9]]*\).*/\1/p'], [0], [dnl
n_bytes=138
])
@@ -1834,7 +1836,11 @@ AT_CHECK([ovs-ofctl dump-flows br0 | grep
"in_port=4" | ofctl_strip], [0], [dnl
n_packets=1, n_bytes=100, priority=1,ip,in_port=4 actions=drop
])
-OVS_TRAFFIC_VSWITCHD_STOP
+OVS_TRAFFIC_VSWITCHD_STOP(["dnl
+/.*lost packet on port.*/d
+/.failed to flow_get.*/d
+/.failed to flow_del.*/d
+/.*Failed to acquire udpif_key corresponding to unexpected flow.*/d"])
AT_CLEANUP
It seems that the overflow here is not related to the race in revalidators,
but it is related to the design of
the testsuite and also the missing of the support of NA/ND in kernel
datapath.
Eelco Chaudron 于2022年9月30日周五 23:47写道：
> On 30 Sep 2022, at 17:41, Peng He wrote:
>
> > It's so easy to reproduce ?
> >
> > Thanks! then I should have to dig it again.
>
> It’s been on my to-do for a while but did not get to it. Sometimes it
> reproduced easily, and sometimes it takes 100+ runs :(
>
> If you apply the following series:
>
> https://patchwork.ozlabs.org/project/openvswitch/list/?series=316861
>
> And then the diff below you can run something like:
>
> sudo bash -c 'for i in {1..100}; do make check-offloads
> TESTSUITEFLAGS="49 50"
>
> 49: datapath - truncate and output to gre tunnel by simulated packets ok
> 50: datapath - truncate and output to gre tunnelok
>
> And after a while, it will fail with the error (sometimes also 0 bytes, I
> hope it’s the same issue :):
>
> system-traffic.at:1663: wait failed after 30 seconds
> n_bytes=18446744073709393054
>
>
>
> $ git diff
> diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
> index 57f94df54..52d61d0a4 100644
> --- a/ofproto/ofproto-dpif-upcall.c
> +++ b/ofproto/ofproto-dpif-upcall.c
> @@ -1868,6 +1868,7 @@ try_ukey_replace(struct umap *umap, struct udpif_key
> *old_ukey,
> ovs_mutex_lock(_ukey->mutex);
> cmap_replace(>cmap, _ukey->cmap_node,
> _ukey->cmap_node, new_ukey->hash);
> +new_ukey->dump_seq = old_ukey->dump_seq;
> ovsrcu_postpone(ukey_delete__, old_uk
Re: [ovs-dev] [ovs-dev v3 3/4] ofproto-dpif-upcall: new ukey needs to take the old ukey's dump seq
Hi, Eelco
I am trying to reproduce the issues.
And I found every time I am running the test case #50, it will
fail at below check:
AT_CHECK([ovs-ofctl dump-flows br0 | grep "in_port=4" | ofctl_strip], [0],
[dnl
n_packets=1, n_bytes=100, priority=1,ip,in_port=4 actions=drop
])
- n_packets=1, n_bytes=100, priority=1,ip,in_port=4 actions=drop
+ priority=1,ip,in_port=4 actions=drop
It looks like it gets zero stats. I am not sure if this fits in the fact
you mentioned sometimes it get 0 stats.
After I did some test, I found that, this is due to the fact that the
environment setting is not correct:
you can try below diff, and test
diff --git a/tests/system-traffic.at b/tests/system-traffic.at
index dfbc30e47..fadeeac29 100644
--- a/tests/system-traffic.at
+++ b/tests/system-traffic.at
@@ -1771,6 +1771,8 @@ AT_CHECK([ovs-ofctl add-flows br0 flows.txt])
AT_CHECK([ovs-ofctl del-flows br-underlay])
AT_DATA([flows-underlay.txt], [dnl
+priority=99,arp,in_port=1,actions=LOCAL
+priority=99,arp,in_port=LOCAL,actions=1
priority=99,dl_type=0x0800,nw_proto=47,in_port=1,actions=LOCAL
priority=99,dl_type=0x0800,nw_proto=47,in_port=LOCAL,ip_dst=
172.31.1.1/24,actions=1
priority=1,actions=drop
Eelco Chaudron 于2022年9月30日周五 23:47写道：
> On 30 Sep 2022, at 17:41, Peng He wrote:
>
> > It's so easy to reproduce ?
> >
> > Thanks! then I should have to dig it again.
>
> It’s been on my to-do for a while but did not get to it. Sometimes it
> reproduced easily, and sometimes it takes 100+ runs :(
>
> If you apply the following series:
>
> https://patchwork.ozlabs.org/project/openvswitch/list/?series=316861
>
> And then the diff below you can run something like:
>
> sudo bash -c 'for i in {1..100}; do make check-offloads
> TESTSUITEFLAGS="49 50"
>
> 49: datapath - truncate and output to gre tunnel by simulated packets ok
> 50: datapath - truncate and output to gre tunnelok
>
> And after a while, it will fail with the error (sometimes also 0 bytes, I
> hope it’s the same issue :):
>
> system-traffic.at:1663: wait failed after 30 seconds
> n_bytes=18446744073709393054
>
>
>
> $ git diff
> diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
> index 57f94df54..52d61d0a4 100644
> --- a/ofproto/ofproto-dpif-upcall.c
> +++ b/ofproto/ofproto-dpif-upcall.c
> @@ -1868,6 +1868,7 @@ try_ukey_replace(struct umap *umap, struct udpif_key
> *old_ukey,
> ovs_mutex_lock(_ukey->mutex);
> cmap_replace(>cmap, _ukey->cmap_node,
> _ukey->cmap_node, new_ukey->hash);
> +new_ukey->dump_seq = old_ukey->dump_seq;
> ovsrcu_postpone(ukey_delete__, old_ukey);
> transition_ukey(old_ukey, UKEY_DELETED);
> transition_ukey(new_ukey, UKEY_VISIBLE);
> diff --git a/tests/system-offloads-testsuite.at b/tests/
> system-offloads-testsuite.at
> index 318e6d1e6..4d546011d 100644
> --- a/tests/system-offloads-testsuite.at
> +++ b/tests/system-offloads-testsuite.at
> @@ -114,8 +114,8 @@ conntrack - DNAT load balancing with NC
>
> # Occasionalt we fail with extreme high byte counters, i.e.
> # n_bytes=18446744073705804134
> -datapath - truncate and output to gre tunnel by simulated packets
> -datapath - truncate and output to gre tunnel
> +#datapath - truncate and output to gre tunnel by simulated packets
> +#datapath - truncate and output to gre tunnel
> "
> echo "$ovs_test_skip_list" | sed "s// /g"])
>
> diff --git a/tests/system-traffic.at b/tests/system-traffic.at
> index 528d2ca64..e86699d1d 100644
> --- a/tests/system-traffic.at
> +++ b/tests/system-traffic.at
> @@ -1708,7 +1708,10 @@ OVS_REVALIDATOR_PURGE()
> OVS_WAIT_UNTIL([ovs-ofctl dump-flows br0 | grep "in_port=4" | ofctl_strip
> | grep "n_packets=1, n_bytes=100, priority=1,ip,in_port=4 actions=drop"],
> [ovs-ofctl dump-flows br0 | grep "in_port=4" |
> ofctl_strip])
>
> -OVS_TRAFFIC_VSWITCHD_STOP
> +OVS_TRAFFIC_VSWITCHD_STOP(["dnl
> +/.*lost packet on handler.*/d
> +/.failed to flow_get.*/d
> +/.*Failed to acquire udpif_key corresponding to unexpected flow.*/d"])
> AT_CLEANUP
>
> dnl Create 2 bridges and 2 namespaces to test truncate over
> @@ -1834,7 +1837,10 @@ AT_CHECK([ovs-ofctl dump-flows br0 | grep
> "in_port=4" | ofctl_strip], [0], [dnl
> n_packets=1, n_bytes=100, priority=1,ip,in_port=4 actions=drop
> ])
>
> -OVS_TRAFFIC_VSWITCHD_STOP
> +OVS_TRAFFIC_VSWITCHD_STOP(["dnl
> +/.*lost packet on handler.*/d
> +/.failed to flow_get.*/d
> +/.*Failed to acquire udpif_key corresponding to unexpected flow.*/d"])
> AT_CLEANUP
>
> AT_SETUP([
Re: [ovs-dev] [ovs-dev v3 4/4] dpif-netdev: fix inconsistent processing between ukey and megaflow
Sure, will do that.
Eelco Chaudron 于2022年10月1日 周六00:03写道：
>
>
> On 30 Sep 2022, at 17:55, Peng He wrote:
>
> > Eelco Chaudron 于2022年9月30日 周五23:50写道：
> >
> >>
> >>
> >> On 23 Sep 2022, at 18:29, Peng He wrote:
> >>
> >>> When PMDs perform upcalls, the newly generated ukey will replace
> >>> the old, however, the newly generated mageflow will be discard
> >>> to reuse the old one without checking if the actions of new and
> >>> old are equal.
> >>>
> >>> This code prevents in case someone runs dpctl/add-flow to add
> >>> a dp flow with inconsistent actions with the actions of ukey,
> >>> and causes more confusion.
> >>>
> >>> Signed-off-by: Peng He
> >>> ---
> >>> lib/dpif-netdev.c | 17 -
> >>> 1 file changed, 16 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> >>> index a45b46014..b316e59ef 100644
> >>> --- a/lib/dpif-netdev.c
> >>> +++ b/lib/dpif-netdev.c
> >>> @@ -8304,7 +8304,22 @@ handle_packet_upcall(struct dp_netdev_pmd_thread
> >> *pmd,
> >>> * to be locking revalidators out of making flow
> modifications.
> >> */
> >>> ovs_mutex_lock(>flow_mutex);
> >>> netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> >>> -if (OVS_LIKELY(!netdev_flow)) {
> >>> +if (OVS_UNLIKELY(netdev_flow)) {
> >>> +struct dp_netdev_actions *old_act =
> >>> +dp_netdev_flow_get_actions(netdev_flow);
> >>> +
> >>> +if ((add_actions->size != old_act->size) ||
> >>> +memcmp(old_act->actions, add_actions->data,
> >>> + add_actions->size)) {
> >>> +
> >>> + struct dp_netdev_actions *new_act =
> >>> + dp_netdev_actions_create(add_actions->data,
> >>> +add_actions->size);
> >>> +
> >>> + ovsrcu_set(_flow->actions, new_act);
> >>> + ovsrcu_postpone(dp_netdev_actions_free, old_act);
> >>
> >> Do we really need this patch? Asking as this is in the fastpath code?
> >> And it only happens if someone is messing with dpctl commands, which
> they
> >> should not.
> >
> >
> > Yes, I just point out there is inconsistent here，for most cases the code
> > will never run into here.
>
> Ok, please add a comment explain this in the next version.
>
> >>
> >> I we want it, we should add a good comment on why it’s there.
> >
> >
> >
> >
> >>
> >>> +}
> >>> +} else {
> >>> netdev_flow = dp_netdev_flow_add(pmd, , ,
> >>> add_actions->data,
> >>> add_actions->size,
> >> orig_in_port);
> >>> --
> >>> 2.25.1
> >>>
> >>> ___
> >>> dev mailing list
> >>> d...@openvswitch.org
> >>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >>
> >> --
> > hepeng
>
> --
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v3 4/4] dpif-netdev: fix inconsistent processing between ukey and megaflow
Eelco Chaudron 于2022年9月30日 周五23:50写道：
>
>
> On 23 Sep 2022, at 18:29, Peng He wrote:
>
> > When PMDs perform upcalls, the newly generated ukey will replace
> > the old, however, the newly generated mageflow will be discard
> > to reuse the old one without checking if the actions of new and
> > old are equal.
> >
> > This code prevents in case someone runs dpctl/add-flow to add
> > a dp flow with inconsistent actions with the actions of ukey,
> > and causes more confusion.
> >
> > Signed-off-by: Peng He
> > ---
> > lib/dpif-netdev.c | 17 -
> > 1 file changed, 16 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> > index a45b46014..b316e59ef 100644
> > --- a/lib/dpif-netdev.c
> > +++ b/lib/dpif-netdev.c
> > @@ -8304,7 +8304,22 @@ handle_packet_upcall(struct dp_netdev_pmd_thread
> *pmd,
> > * to be locking revalidators out of making flow modifications.
> */
> > ovs_mutex_lock(>flow_mutex);
> > netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL);
> > -if (OVS_LIKELY(!netdev_flow)) {
> > +if (OVS_UNLIKELY(netdev_flow)) {
> > +struct dp_netdev_actions *old_act =
> > +dp_netdev_flow_get_actions(netdev_flow);
> > +
> > +if ((add_actions->size != old_act->size) ||
> > +memcmp(old_act->actions, add_actions->data,
> > + add_actions->size)) {
> > +
> > + struct dp_netdev_actions *new_act =
> > + dp_netdev_actions_create(add_actions->data,
> > +add_actions->size);
> > +
> > + ovsrcu_set(_flow->actions, new_act);
> > + ovsrcu_postpone(dp_netdev_actions_free, old_act);
>
> Do we really need this patch? Asking as this is in the fastpath code?
> And it only happens if someone is messing with dpctl commands, which they
> should not.
Yes, I just point out there is inconsistent here，for most cases the code
will never run into here.
>
> I we want it, we should add a good comment on why it’s there.
>
> > +}
> > +} else {
> > netdev_flow = dp_netdev_flow_add(pmd, , ,
> > add_actions->data,
> > add_actions->size,
> orig_in_port);
> > --
> > 2.25.1
> >
> > ___
> > dev mailing list
> > d...@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
> --
hepeng
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [ovs-dev v3 3/4] ofproto-dpif-upcall: new ukey needs to take the old ukey's dump seq
thanks a lot, appreciate it!
Eelco Chaudron 于2022年9月30日周五 23:47写道：
> On 30 Sep 2022, at 17:41, Peng He wrote:
>
> > It's so easy to reproduce ?
> >
> > Thanks! then I should have to dig it again.
>
> It’s been on my to-do for a while but did not get to it. Sometimes it
> reproduced easily, and sometimes it takes 100+ runs :(
>
> If you apply the following series:
>
> https://patchwork.ozlabs.org/project/openvswitch/list/?series=316861
>
> And then the diff below you can run something like:
>
> sudo bash -c 'for i in {1..100}; do make check-offloads
> TESTSUITEFLAGS="49 50"
>
> 49: datapath - truncate and output to gre tunnel by simulated packets ok
> 50: datapath - truncate and output to gre tunnelok
>
> And after a while, it will fail with the error (sometimes also 0 bytes, I
> hope it’s the same issue :):
>
> system-traffic.at:1663: wait failed after 30 seconds
> n_bytes=18446744073709393054
>
>
>
> $ git diff
> diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
> index 57f94df54..52d61d0a4 100644
> --- a/ofproto/ofproto-dpif-upcall.c
> +++ b/ofproto/ofproto-dpif-upcall.c
> @@ -1868,6 +1868,7 @@ try_ukey_replace(struct umap *umap, struct udpif_key
> *old_ukey,
> ovs_mutex_lock(_ukey->mutex);
> cmap_replace(>cmap, _ukey->cmap_node,
> _ukey->cmap_node, new_ukey->hash);
> +new_ukey->dump_seq = old_ukey->dump_seq;
> ovsrcu_postpone(ukey_delete__, old_ukey);
> transition_ukey(old_ukey, UKEY_DELETED);
> transition_ukey(new_ukey, UKEY_VISIBLE);
> diff --git a/tests/system-offloads-testsuite.at b/tests/
> system-offloads-testsuite.at
> index 318e6d1e6..4d546011d 100644
> --- a/tests/system-offloads-testsuite.at
> +++ b/tests/system-offloads-testsuite.at
> @@ -114,8 +114,8 @@ conntrack - DNAT load balancing with NC
>
> # Occasionalt we fail with extreme high byte counters, i.e.
> # n_bytes=18446744073705804134
> -datapath - truncate and output to gre tunnel by simulated packets
> -datapath - truncate and output to gre tunnel
> +#datapath - truncate and output to gre tunnel by simulated packets
> +#datapath - truncate and output to gre tunnel
> "
> echo "$ovs_test_skip_list" | sed "s// /g"])
>
> diff --git a/tests/system-traffic.at b/tests/system-traffic.at
> index 528d2ca64..e86699d1d 100644
> --- a/tests/system-traffic.at
> +++ b/tests/system-traffic.at
> @@ -1708,7 +1708,10 @@ OVS_REVALIDATOR_PURGE()
> OVS_WAIT_UNTIL([ovs-ofctl dump-flows br0 | grep "in_port=4" | ofctl_strip
> | grep "n_packets=1, n_bytes=100, priority=1,ip,in_port=4 actions=drop"],
> [ovs-ofctl dump-flows br0 | grep "in_port=4" |
> ofctl_strip])
>
> -OVS_TRAFFIC_VSWITCHD_STOP
> +OVS_TRAFFIC_VSWITCHD_STOP(["dnl
> +/.*lost packet on handler.*/d
> +/.failed to flow_get.*/d
> +/.*Failed to acquire udpif_key corresponding to unexpected flow.*/d"])
> AT_CLEANUP
>
> dnl Create 2 bridges and 2 namespaces to test truncate over
> @@ -1834,7 +1837,10 @@ AT_CHECK([ovs-ofctl dump-flows br0 | grep
> "in_port=4" | ofctl_strip], [0], [dnl
> n_packets=1, n_bytes=100, priority=1,ip,in_port=4 actions=drop
> ])
>
> -OVS_TRAFFIC_VSWITCHD_STOP
> +OVS_TRAFFIC_VSWITCHD_STOP(["dnl
> +/.*lost packet on handler.*/d
> +/.failed to flow_get.*/d
> +/.*Failed to acquire udpif_key corresponding to unexpected flow.*/d"])
> AT_CLEANUP
>
> AT_SETUP([datapath - configure cache size])
>
>
>
> > Eelco Chaudron 于2022年9月30日周五 23:39写道：
> >
> >>
> >>
> >> On 30 Sep 2022, at 17:26, Peng He wrote:
> >>
> >>> Eelco Chaudron 于2022年9月30日周五 23:01写道：
> >>>
> >>>>
> >>>>
> >>>> On 23 Sep 2022, at 18:29, Peng He wrote:
> >>>>
> >>>>> The userspace datapath mananges all the magaflows by a cmap. The cmap
> >>>>> data structrue will grow/shrink during the datapath processing and it
> >>>>> will re-position megaflows. This might result in two revalidator
> >> threads
> >>>>> might process a same megaflow during one dump stage.
> >>>>>
> >>>>> Consider a situation that, revalidator 1 processes a megaflow A, and
> >>>>> decides to delete it from the datapath, at the mean time, this
> megaflow
> >>>>> A is also queued in the process batch of revalidator 2. Normally it's
> >> ok
> >>>>> for revalidators to process the same meg
Re: [ovs-dev] [ovs-dev v3 3/4] ofproto-dpif-upcall: new ukey needs to take the old ukey's dump seq
can you show me how to reproduce the case? since it is easy to reproduce, and we need openflow stats for billing case, so it's important to keep it accurate. Thanks! Eelco Chaudron 于2022年9月30日周五 23:39写道： > > > On 30 Sep 2022, at 17:26, Peng He wrote: > > > Eelco Chaudron 于2022年9月30日周五 23:01写道： > > > >> > >> > >> On 23 Sep 2022, at 18:29, Peng He wrote: > >> > >>> The userspace datapath mananges all the magaflows by a cmap. The cmap > >>> data structrue will grow/shrink during the datapath processing and it > >>> will re-position megaflows. This might result in two revalidator > threads > >>> might process a same megaflow during one dump stage. > >>> > >>> Consider a situation that, revalidator 1 processes a megaflow A, and > >>> decides to delete it from the datapath, at the mean time, this megaflow > >>> A is also queued in the process batch of revalidator 2. Normally it's > ok > >>> for revalidators to process the same megaflow multiple times, as the > >>> dump_seq shows it's already dumped and the stats will not be > contributed > >>> twice. > >>> > >>> Assume that right after A is deleted, a PMD thread generates again > >>> a new megaflow B which has the same match and action of A. The ukey > >>> of megaflow B will replace the one of megaflow A. Now the ukey B is > >>> new to the revalidator system and its dump seq is 0. > >>> > >>> Now since the dump seq of ukey B is 0, when processing megaflow A, > >>> the revalidator 2 will not identify this megaflow A has already been > >>> dumped by revalidator 1 and will contribute the old megaflow A's stats > >>> again, this results in an inconsistent stats between ukeys and > megaflows. > >>> > >>> To fix this, the newly generated the ukey B should take the dump_seq > >>> of the replaced ukey A to avoid a same megaflow being revalidated > >>> twice in one dump stage. > >>> > >>> We observe in the production environment, the OpenFlow rules' stats > >>> sometimes are amplified compared to the actual value. I believe this > >>> is also the reason that why somtimes there is mismatch between the > >>> ukey and megaflow in stats value. The Eelco's patch > >>> [ovs-dev] [PATCH v2 09/10] revalidator: Fix datapath statistics update > >>> tried to fix it in the past. > >> > >> This sounds plausible, are your statistics extremely elevated? > >> Mine are in the likes of n_bytes=18446744073705804134 where it should be > >> around 100. > >> > >> It looks more like an overflow. > > I just get the report from another team, I need to ask them, we will > have a > > 7 days off due to the national day. > > so it will take time to get the value. :( > > Enjoy your time off!! This fix is not solving my problem; > > Still get the error once out of X runs, n_bytes=18446744073709393054 > > > I’ll try to get my old setup up and run it continuously over the weekend > >> and see if it’s replicated again. > >> > >> thanks! > > > > > >>> Signed-off-by: Peng He > >>> --- > >>> ofproto/ofproto-dpif-upcall.c | 1 + > >>> 1 file changed, 1 insertion(+) > >>> > >>> diff --git a/ofproto/ofproto-dpif-upcall.c > >> b/ofproto/ofproto-dpif-upcall.c > >>> index e8bbcfeaf..89fad1bdf 100644 > >>> --- a/ofproto/ofproto-dpif-upcall.c > >>> +++ b/ofproto/ofproto-dpif-upcall.c > >>> @@ -1877,6 +1877,7 @@ try_ukey_replace(struct umap *umap, struct > >> udpif_key *old_ukey, > >>> ovs_mutex_lock(_ukey->mutex); > >>> cmap_replace(>cmap, _ukey->cmap_node, > >>> _ukey->cmap_node, new_ukey->hash); > >>> +new_ukey->dump_seq = old_ukey->dump_seq; > >>> ovsrcu_postpone(ukey_delete__, old_ukey); > >>> transition_ukey(old_ukey, UKEY_DELETED); > >>> transition_ukey(new_ukey, UKEY_VISIBLE); > >>> -- > >>> 2.25.1 > >>> > >>> ___ > >>> dev mailing list > >>> d...@openvswitch.org > >>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev > >> > >> > > > > -- > > hepeng > > -- hepeng ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
