On 13 Oct 2026, at 1:46, Ilya Maximets via dev wrote:
> On 7/6/26 5:27 PM, Eelco Chaudron via dev wrote:
>> When tc_netdev_flow_put() rejects a flow because the upstream
>> recirc chain is not yet registered (used_chains count is zero),
>> the flow is permanently stranded in the kernel. Return EAGAIN
>> instead of EOPNOTSUPP so the revalidator retries the offload
>> once a sibling flow registers the chain.
>>
>> A new bool offload_deferred on struct dpif_op and struct udpif_key
>> propagates the rejection to the revalidator, which issues a retry
>> on the next cycle. Retries are bounded to
>> OFFLOAD_DEFERRED_RETRY_MSEC (2000ms) from the first failed attempt.
>>
>> Fixes: 273a4fce951a ("netdev-offload-tc: Only install recirc flows if the
>> parent is present.")
>> Reported-by: Numan Siddique <[email protected]>
>> Signed-off-by: Eelco Chaudron <[email protected]>
>> ---
>> lib/dpif-offload-tc-netdev.c | 27 +++++++------
>> lib/dpif-offload-tc.c | 10 ++++-
>> lib/dpif.c | 7 ++++
>> lib/dpif.h | 3 ++
>> ofproto/ofproto-dpif-upcall.c | 27 +++++++++++++
>> tests/system-offloads-traffic.at | 68 ++++++++++++++++++++++++++++++++
>> 6 files changed, 129 insertions(+), 13 deletions(-)
>
> Hi, Eelco and Numan. Thanks for working on this and sorry for delay.
Thanks for the review and the suggestions. See my reply below; however,
I will send out a v2 soon.
//Eelco
>
> I have a few concerns about this solution:
>
> 1. The EAGAIN may be returned for different cases, not just the missing
> chain, but the code and the comments in the generic code above seems
> to assume that. Maybe we should introduce a flag for the info
> structure and pass it up from the tc_netdev_flow_put() while keep
> returning EOPNOTSUPP?
ACK; will replace EAGAIN with an offload_deferred flag on
tc_offload_info and keep returning EOPNOTSUPP in v2.
> 2. The time-based re-tries do not seem robust enough. Various operations
> could take more than two seconds under the high load and we may miss
> the window to re-try. Maybe a better solution would be to expose a
> sequence number from the dpif-offload module that providers can bump
> when they anticipate potential change in the offloadability? E.g.
> the tc provider could bump this sequence number when ccmap_inc()
> returns one, i.e. when the new chain is used. The revalidators could
> then re-try deferred flows that have lower sequence number and update
> it when it still fails, so it's not retried more than once per change.
> We would still probbaly want a global limit, so we do not re-try
> forever, but maybe someting like a flat attmepts counter per ukey
> would suffice, e.g. no more than 16 attempts, or something like this.
This is a good suggestion. I'll work on this, and sent out a v2.
> What do you think?
>
> A few more comments below.
>
>>
>> diff --git a/lib/dpif-offload-tc-netdev.c b/lib/dpif-offload-tc-netdev.c
>> index a2f4ab8b5..3a521aee8 100644
>> --- a/lib/dpif-offload-tc-netdev.c
>> +++ b/lib/dpif-offload-tc-netdev.c
>> @@ -2355,17 +2355,6 @@ tc_netdev_flow_put(struct dpif *dpif, struct netdev
>> *netdev,
>> chain = key->recirc_id;
>> mask->recirc_id = 0;
>>
>> - if (chain) {
>> - /* If we match on a recirculation ID, we must ensure the previous
>> - * flow is also in the TC datapath; otherwise, the entry is useless,
>> - * as the related packets will be handled by upcalls. */
>> - if (!ccmap_find(&used_chains, chain)) {
>> - VLOG_DBG_RL(&rl, "match for chain %u failed due to non-existing
>> "
>> - "goto chain action", chain);
>> - return EOPNOTSUPP;
>> - }
>> - }
>> -
>> if (flow_tnl_dst_is_set(&key->tunnel) ||
>> flow_tnl_src_is_set(&key->tunnel)) {
>> VLOG_DBG_RL(&rl,
>> @@ -2687,6 +2676,22 @@ tc_netdev_flow_put(struct dpif *dpif, struct netdev
>> *netdev,
>> return EOPNOTSUPP;
>> }
>>
>> + if (chain && !ccmap_find(&used_chains, chain)) {
>> + /* If we match on a recirculation ID, we must ensure the previous
>> + * flow is also in the TC datapath; otherwise, the entry is useless,
>> + * as the related packets will be handled by upcalls.
>> + *
>> + * Return EAGAIN rather than EOPNOTSUPP: the chain may not be
>> + * registered yet because the upstream flow install is still in
>> + * progress. Unlike EOPNOTSUPP, EAGAIN tells the revalidator to
>> + * retry the TC offload on a subsequent cycle. The flow is still
>> + * installed in the kernel datapath (dp:ovs) in either case. */
>
> This is not always the case, right? Only in case of AUTO.
You are right, only true for DPIF_OFFLOAD_AUTO. Will remove
these details to avoid confusion.
>> + VLOG_DBG_RL(&rl, "match for chain %u failed due to non-existing "
>> + "goto chain action",
>> + chain);
>
> nit: keep on the same line.
ACK
>> + return EAGAIN;
>> + }
>> +
>> memset(&adjust_stats, 0, sizeof adjust_stats);
>> if (get_ufid_tc_mapping(ufid, &id) == 0) {
>> VLOG_DBG_RL(&rl, "updating old handle: %d prio: %d",
>> diff --git a/lib/dpif-offload-tc.c b/lib/dpif-offload-tc.c
>> index 3b84d9f20..cb996fa4a 100644
>> --- a/lib/dpif-offload-tc.c
>> +++ b/lib/dpif-offload-tc.c
>> @@ -579,7 +579,8 @@ tc_parse_flow_put(struct tc_offload *offload_tc, struct
>> dpif *dpif,
>> }
>> netdev_set_hw_info(oor_netdev, HW_INFO_TYPE_OOR, true);
>> }
>> - level = (err == ENOSPC || err == EOPNOTSUPP) ? VLL_DBG : VLL_ERR;
>> + level = (err == ENOSPC || err == EOPNOTSUPP
>> + || err == EAGAIN) ? VLL_DBG : VLL_ERR;
>> VLOG_RL(&rl, level, "failed to offload flow: %s: %s",
>> ovs_strerror(err),
>> (oor_netdev ? netdev_get_name(oor_netdev) :
>> @@ -705,7 +706,12 @@ tc_operate(struct dpif *dpif, const struct dpif_offload
>> *offload_,
>> break;
>> } /* End of 'switch (op->type)'. */
>>
>> - if (error != EOPNOTSUPP && error != ENOENT) {
>> + if (error == EAGAIN) {
>> + /* The recirc ID matched by this flow is not yet registered in
>> + * TC (no upstream flow has a goto action for this chain).
>> + * Defer the offload so the revalidator retries next cycle. */
>> + op->offload_deferred = true;
>> + } else if (error != EOPNOTSUPP && error != ENOENT) {
>> /* If the operation is unsupported or the entry was not found,
>> * we are skipping this flow operation. Otherwise, it was
>> * processed and we should report the result. */
>> diff --git a/lib/dpif.c b/lib/dpif.c
>> index 1afc7c662..0b93b0fae 100644
>> --- a/lib/dpif.c
>> +++ b/lib/dpif.c
>> @@ -1354,10 +1354,17 @@ dpif_operate(struct dpif *dpif, struct dpif_op
>> **ops, size_t n_ops,
>> for (i = 0; i < n_ops; i++) {
>> struct dpif_op *op = ops[i];
>> op->error = EINVAL;
>> + op->offload_deferred = false;
>> }
>> return;
>> }
>>
>> + /* Initialize offload_deferred for all ops; the offload provider sets
>> + * it to true if the flow cannot be offloaded yet. */
>> + for (size_t i = 0; i < n_ops; i++) {
>> + ops[i]->offload_deferred = false;
>> + }
>> +
>> while (n_ops > 0) {
>> size_t chunk;
>>
>> diff --git a/lib/dpif.h b/lib/dpif.h
>> index 3e6a34a25..79621887c 100644
>> --- a/lib/dpif.h
>> +++ b/lib/dpif.h
>> @@ -782,6 +782,9 @@ int dpif_execute(struct dpif *, struct dpif_execute *);
>> struct dpif_op {
>> enum dpif_op_type type;
>> int error;
>> + bool offload_deferred; /* True if the offload provider requires a
>> + * prerequisite flow to be programmed first.
>> + * The revalidator will retry on the next cycle.
>> */
>> union {
>> struct dpif_flow_put flow_put;
>> struct dpif_flow_del flow_del;
>> diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
>> index 8e4897202..85071e852 100644
>> --- a/ofproto/ofproto-dpif-upcall.c
>> +++ b/ofproto/ofproto-dpif-upcall.c
>> @@ -50,6 +50,7 @@
>> #define UPCALL_MAX_BATCH 64
>> #define REVALIDATE_MAX_BATCH 50
>> #define UINT64_THREE_QUARTERS (UINT64_MAX / 4 * 3)
>> +#define OFFLOAD_DEFERRED_RETRY_MSEC 2000
>>
>> VLOG_DEFINE_THIS_MODULE(ofproto_dpif_upcall);
>>
>> @@ -347,6 +348,8 @@ struct udpif_key {
>> #define OFFL_REBAL_INTVL_MSEC 3000 /* dynamic offload rebalance freq */
>> struct netdev *in_netdev; /* in_odp_port's netdev */
>> bool offloaded; /* True if flow is offloaded */
>> + bool offload_deferred; /* Deferred, retry next cycle. */
>> + long long int offload_deferred_expire; /* Retry deadline in msec. */
>> uint64_t flow_pps_rate; /* Packets-Per-Second rate */
>> long long int flow_time; /* last pps update time */
>> uint64_t flow_packets; /* #pkts seen in interval */
>> @@ -1746,6 +1749,11 @@ handle_upcalls(struct udpif *udpif, struct upcall
>> *upcalls,
>> transition_ukey(ukey, UKEY_EVICTED);
>> } else if (ukey->state < UKEY_OPERATIONAL) {
>> transition_ukey(ukey, UKEY_OPERATIONAL);
>> + if (ops[i].dop.offload_deferred) {
>> + ukey->offload_deferred_expire =
>> + time_msec() + OFFLOAD_DEFERRED_RETRY_MSEC;
>> + ukey->offload_deferred = true;
>> + }
>
> May it be possible that revlidator move it into operation state already?
Good catch; the revalidator can race and promote the ukey
to OPERATIONAL before we re-lock, so the deferred flag is
never set. Will fix it in the next version.
> Best regards, Ilya Maximets.
> _______________________________________________
> dev mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev