On 23-10-2026 01:12, Haiyang Zhang wrote: > > >> -----Original Message----- >> From: Sahil Chandna <[email protected]> >> Sent: Thursday, October 22, 2026 10:54 AM >> To: Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui >> <[email protected]>; [email protected]; [email protected]; >> [email protected]; [email protected]; [email protected]; Konstantin >> Taranov <[email protected]>; [email protected]; >> [email protected]; [email protected]; >> [email protected]; [email protected]; linux- >> [email protected]; [email protected]; linux- >> [email protected]; [email protected] >> Cc: [email protected] >> Subject: [PATCH net v3] net: mana: Add support to process throttled EQEs >> >> When an event queue nears full, the hardware coalesces the per-CQ >> GDMA_EQE_COMPLETION notifications into a single throttle EQE (type 4) >> that carries no cq_id. The driver does not handle this event type, so >> it falls through to the default case and is dropped. Any CQ whose >> completion notification was replaced by the throttle EQE can remain >> unserviced, causing its queue to stall. >> The issue was observed internally and following Tx stall logs were >> seen: >> >> 2026-10-03T05:29:01 hv_netvsc eth0: VF slot 0 removed >> 2026-10-03T05:29:23 mana 7870:00:00.0 ens1: NETDEV WATCHDOG: CPU: 45: >> transmit queue 1 timed out 15133 ms >> 2026-10-03T05:31:05 mana 7870:00:00.0: HWC: Request timed out: 60000 ms >> 2026-10-03T05:31:05 mana 7870:00:00.0: Gf stats wk handler: gf stats >> query timed out. >> 2026-10-03T05:31:21 ens1: NETDEV WATCHDOG: CPU: 20: transmit queue 0 >> timed out 15859 ms >> 2026-10-03T05:31:37 ens1: NETDEV WATCHDOG: CPU: 15: transmit queue 0 >> timed out 31731 ms >> 2026-10-03T05:31:53 ens1: NETDEV WATCHDOG: CPU: 21: transmit queue 0 >> timed out 47603 ms >> 2026-10-03T05:32:08 ens1: NETDEV WATCHDOG: CPU: 42: transmit queue 0 >> timed out 62451 ms >> >> Add support to process the CQs which belong to a throttled EQE of type 4 >> advertise the capability bit, also add a counter to store the throttled >> EQEs count. >> >> Fixes: ca9c54d2d6a5 ("net: mana: Add a driver for Microsoft Azure Network >> Adapter (MANA)") >> Cc: [email protected] >> Signed-off-by: Sahil Chandna <[email protected]> >> --- >> Changes in v3: >> - Resolve race for cq in mana_gd_destroy_cq and irq handler using cq. >> - Rebase to net branch >> Link to v2: >> https://lore.ker/ >> nel.org%2Fall%2F20261014134304.4177657-1- >> sahilchandna%40linux.microsoft.com%2F&data=05%7C02%7Chaiyangz%40microsoft. >> com%7C534992a49c0340631e2f08df304e3c73%7C72f988bf86f141af91ab2d7cd011db47% >> 7C1%7C0%7C639282784768349454%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRyd >> WUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C >> 0%7C%7C%7C&sdata=w2NpUCx%2FCNtqeFQSR0fIMOYnbHLP7XsFl90sL%2F%2F7Y0g%3D&rese >> rved=0 >> >> Changes in v2: >> - Use WRITE_ONCE() when clearing max_num_cqs and add RCU >> grace period wait during HWC teardown. >> - Use synchronize_net() during CQ destroy only when the CQ was >> attached to its parent EQ. >> - Use WRITE_ONCE() and READ_ONCE() for the throttle-event counter. >> - Add Fixes tag and net instead of net-next. >> - Fix commit message to describe capability bit and counter. >> Link to v1: >> https://lore.ker/ >> nel.org%2Fall%2F20261008083315.306076-1- >> sahilchandna%40linux.microsoft.com%2F&data=05%7C02%7Chaiyangz%40microsoft. >> com%7C534992a49c0340631e2f08df304e3c73%7C72f988bf86f141af91ab2d7cd011db47% >> 7C1%7C0%7C639282784768392174%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRyd >> WUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C >> 0%7C%7C%7C&sdata=oj6Ei41iOntNjgoAWNaLSZdvptpbZSkkZMOCa6c2PDw%3D&reserved=0 >> >> .../net/ethernet/microsoft/mana/gdma_main.c | 62 +++++++++++++++++-- >> .../net/ethernet/microsoft/mana/hw_channel.c | 6 +- >> .../ethernet/microsoft/mana/mana_ethtool.c | 9 +++ >> include/net/mana/gdma.h | 12 +++- >> include/net/mana/mana.h | 1 + >> 5 files changed, 81 insertions(+), 9 deletions(-) >> >> diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c >> b/drivers/net/ethernet/microsoft/mana/gdma_main.c >> index 8e9bfc1d6a2a..d74ed00bb783 100644 >> --- a/drivers/net/ethernet/microsoft/mana/gdma_main.c >> +++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c >> @@ -904,6 +904,27 @@ ssize_t mana_gd_read_ring(struct gdma_queue *q, char >> __user *buf, >> return copied; >> } >> >> +static void mana_gd_schedule_eq_cqs(struct gdma_queue *eq) >> +{ >> + struct gdma_context *gc = eq->gdma_dev->gdma_context; >> + struct gdma_queue *cq; >> + u8 i; >> + >> + for (i = 0; i < GDMA_EQ_MAX_CHILD_CQ; i++) { >> + cq = rcu_dereference(eq->eq.child_cq[i]); >> + >> + if (!cq) >> + continue; >> + >> + if (cq->id >= READ_ONCE(gc->max_num_cqs) || >> + !gc->cq_table || gc->cq_table[cq->id] != cq) >> + continue; >> + >> + if (cq->cq.callback) >> + cq->cq.callback(cq->cq.context, cq); >> + } >> +} >> + >> static void mana_gd_process_eqe(struct gdma_queue *eq) >> { >> u32 head = eq->head % (eq->queue_size / GDMA_EQE_SIZE); >> @@ -934,6 +955,11 @@ static void mana_gd_process_eqe(struct gdma_queue >> *eq) >> >> break; >> >> + case GDMA_EQE_THROTTLE: >> + WRITE_ONCE(eq->eq.throttle_count, eq->eq.throttle_count + 1); > Please use u64_stats_update_begin / end() for stats, which is the standard > way -- see other stats code in this driver. > Ack, will update it in v4.>> + mana_gd_schedule_eq_cqs(eq); >> + break; >> + >> case GDMA_EQE_TEST_EVENT: >> gc->test_event_eq_id = eq->id; >> complete(&gc->eq_test_event); > [...] > > >> @@ -250,6 +252,7 @@ static void mana_get_ethtool_stats(struct net_device >> *ndev, >> u64 short_pkt_fmt; >> u64 csum_partial; >> u64 mana_map_err; >> + u64 throttle_events; >> int q, i = 0, j; >> >> if (!apc->port_is_up) >> @@ -261,6 +264,12 @@ static void mana_get_ethtool_stats(struct net_device >> *ndev, >> */ >> mana_query_phy_stats(apc); >> >> + throttle_events = 0; >> + for (q = 0; q < num_queues; q++) >> + throttle_events += READ_ONCE(apc->eqs[q].eq- >>> eq.throttle_count); > > Since you already have per queue throttle_count, please output them for > each EQ. > > Thanks, > - Haiyang Ack, i will drop accumulated throttle count and report per EQ in v4.
The patch needs rebase again for net branch, i will rebase in v4.
