> -----Original Message-----
> From: Sahil Chandna <[email protected]>
> Sent: Thursday, October 22, 2026 10:54 AM
> To: Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui
> <[email protected]>; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected]; Konstantin
> Taranov <[email protected]>; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; linux-
> [email protected]; [email protected]; linux-
> [email protected]; [email protected]
> Cc: [email protected]
> Subject: [PATCH net v3] net: mana: Add support to process throttled EQEs
>
> When an event queue nears full, the hardware coalesces the per-CQ
> GDMA_EQE_COMPLETION notifications into a single throttle EQE (type 4)
> that carries no cq_id. The driver does not handle this event type, so
> it falls through to the default case and is dropped. Any CQ whose
> completion notification was replaced by the throttle EQE can remain
> unserviced, causing its queue to stall.
> The issue was observed internally and following Tx stall logs were
> seen:
>
> 2026-10-03T05:29:01 hv_netvsc eth0: VF slot 0 removed
> 2026-10-03T05:29:23 mana 7870:00:00.0 ens1: NETDEV WATCHDOG: CPU: 45:
> transmit queue 1 timed out 15133 ms
> 2026-10-03T05:31:05 mana 7870:00:00.0: HWC: Request timed out: 60000 ms
> 2026-10-03T05:31:05 mana 7870:00:00.0: Gf stats wk handler: gf stats
> query timed out.
> 2026-10-03T05:31:21 ens1: NETDEV WATCHDOG: CPU: 20: transmit queue 0
> timed out 15859 ms
> 2026-10-03T05:31:37 ens1: NETDEV WATCHDOG: CPU: 15: transmit queue 0
> timed out 31731 ms
> 2026-10-03T05:31:53 ens1: NETDEV WATCHDOG: CPU: 21: transmit queue 0
> timed out 47603 ms
> 2026-10-03T05:32:08 ens1: NETDEV WATCHDOG: CPU: 42: transmit queue 0
> timed out 62451 ms
>
> Add support to process the CQs which belong to a throttled EQE of type 4
> advertise the capability bit, also add a counter to store the throttled
> EQEs count.
>
> Fixes: ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network
> Adapter (MANA)")
> Cc: [email protected]
> Signed-off-by: Sahil Chandna <[email protected]>
> ---
> Changes in v3:
> - Resolve race for cq in mana_gd_destroy_cq and irq handler using cq.
> - Rebase to net branch
> Link to v2:
> https://lore.ker/
> nel.org%2Fall%2F20261014134304.4177657-1-
> sahilchandna%40linux.microsoft.com%2F&data=05%7C02%7Chaiyangz%40microsoft.
> com%7C534992a49c0340631e2f08df304e3c73%7C72f988bf86f141af91ab2d7cd011db47%
> 7C1%7C0%7C639282784768349454%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRyd
> WUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C
> 0%7C%7C%7C&sdata=w2NpUCx%2FCNtqeFQSR0fIMOYnbHLP7XsFl90sL%2F%2F7Y0g%3D&rese
> rved=0
>
> Changes in v2:
> - Use WRITE_ONCE() when clearing max_num_cqs and add RCU
> grace period wait during HWC teardown.
> - Use synchronize_net() during CQ destroy only when the CQ was
> attached to its parent EQ.
> - Use WRITE_ONCE() and READ_ONCE() for the throttle-event counter.
> - Add Fixes tag and net instead of net-next.
> - Fix commit message to describe capability bit and counter.
> Link to v1:
> https://lore.ker/
> nel.org%2Fall%2F20261008083315.306076-1-
> sahilchandna%40linux.microsoft.com%2F&data=05%7C02%7Chaiyangz%40microsoft.
> com%7C534992a49c0340631e2f08df304e3c73%7C72f988bf86f141af91ab2d7cd011db47%
> 7C1%7C0%7C639282784768392174%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRyd
> WUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C
> 0%7C%7C%7C&sdata=oj6Ei41iOntNjgoAWNaLSZdvptpbZSkkZMOCa6c2PDw%3D&reserved=0
>
> .../net/ethernet/microsoft/mana/gdma_main.c | 62 +++++++++++++++++--
> .../net/ethernet/microsoft/mana/hw_channel.c | 6 +-
> .../ethernet/microsoft/mana/mana_ethtool.c | 9 +++
> include/net/mana/gdma.h | 12 +++-
> include/net/mana/mana.h | 1 +
> 5 files changed, 81 insertions(+), 9 deletions(-)
>
> diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c
> b/drivers/net/ethernet/microsoft/mana/gdma_main.c
> index 8e9bfc1d6a2a..d74ed00bb783 100644
> --- a/drivers/net/ethernet/microsoft/mana/gdma_main.c
> +++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c
> @@ -904,6 +904,27 @@ ssize_t mana_gd_read_ring(struct gdma_queue *q, char
> __user *buf,
> return copied;
> }
>
> +static void mana_gd_schedule_eq_cqs(struct gdma_queue *eq)
> +{
> + struct gdma_context *gc = eq->gdma_dev->gdma_context;
> + struct gdma_queue *cq;
> + u8 i;
> +
> + for (i = 0; i < GDMA_EQ_MAX_CHILD_CQ; i++) {
> + cq = rcu_dereference(eq->eq.child_cq[i]);
> +
> + if (!cq)
> + continue;
> +
> + if (cq->id >= READ_ONCE(gc->max_num_cqs) ||
> + !gc->cq_table || gc->cq_table[cq->id] != cq)
> + continue;
> +
> + if (cq->cq.callback)
> + cq->cq.callback(cq->cq.context, cq);
> + }
> +}
> +
> static void mana_gd_process_eqe(struct gdma_queue *eq)
> {
> u32 head = eq->head % (eq->queue_size / GDMA_EQE_SIZE);
> @@ -934,6 +955,11 @@ static void mana_gd_process_eqe(struct gdma_queue
> *eq)
>
> break;
>
> + case GDMA_EQE_THROTTLE:
> + WRITE_ONCE(eq->eq.throttle_count, eq->eq.throttle_count + 1);
Please use u64_stats_update_begin / end() for stats, which is the standard
way -- see other stats code in this driver.
> + mana_gd_schedule_eq_cqs(eq);
> + break;
> +
> case GDMA_EQE_TEST_EVENT:
> gc->test_event_eq_id = eq->id;
> complete(&gc->eq_test_event);
[...]
> @@ -250,6 +252,7 @@ static void mana_get_ethtool_stats(struct net_device
> *ndev,
> u64 short_pkt_fmt;
> u64 csum_partial;
> u64 mana_map_err;
> + u64 throttle_events;
> int q, i = 0, j;
>
> if (!apc->port_is_up)
> @@ -261,6 +264,12 @@ static void mana_get_ethtool_stats(struct net_device
> *ndev,
> */
> mana_query_phy_stats(apc);
>
> + throttle_events = 0;
> + for (q = 0; q < num_queues; q++)
> + throttle_events += READ_ONCE(apc->eqs[q].eq-
> >eq.throttle_count);
Since you already have per queue throttle_count, please output them for
each EQ.
Thanks,
- Haiyang
