Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] net: mana: non-waiting CQ poll in mana_dealloc_queues() triggers false FLR - [Medium] net: mana: mana_hwc_send_request() blocks on hwc->sema before checking hwc_dead --
commit 7823c203253d16c1fb640cce922ae30200b9f486 Author: Sahil Chandna <[email protected]> net: mana: Avoid waiting on dead HWC after device reset This commit replaces the sleep-based TX queue drain in mana_dealloc_queues() with synchronous CQE polling and marks the hardware channel dead via hwc_dead after a device reset so subsequent HWC commands fail fast with -ETIMEDOUT. > diff --git a/drivers/net/ethernet/microsoft/mana/hw_channel.c > b/drivers/net/ethernet/microsoft/mana/hw_channel.c > index 3bca4b683134b..c9337f49fbe15 100644 > --- a/drivers/net/ethernet/microsoft/mana/hw_channel.c > +++ b/drivers/net/ethernet/microsoft/mana/hw_channel.c [ ... ] > @@ -862,6 +867,7 @@ int mana_hwc_send_request(struct hw_channel_context *hwc, > u32 req_len, > u32 dest_vrcq; > u32 dest_vrq; > u32 command; > + u32 timeout = 0; > u16 msg_id; > int err; > > @@ -897,6 +903,17 @@ int mana_hwc_send_request(struct hw_channel_context > *hwc, u32 req_len, > dest_vrq = hwc->dest_vrq_id; > dest_vrcq = hwc->dest_vrcq_id; > > + /* If the HWC is unavailable (e.g. hwc_timeout was set to 0 or > + * hwc_dead set after device reset), do not post the WQE: the > + * hardware cannot respond, so fail fast instead of waiting out > + * the full timeout. > + */ > + timeout = READ_ONCE(hwc->hwc_timeout); > + if (timeout == 0 || READ_ONCE(hwc->hwc_dead)) { > + err = -ETIMEDOUT; > + goto out; > + } [Severity: Medium] Can mana_hwc_send_request() still block for the full hwc_timeout on hwc->sema before reaching this hwc_dead check? At the start of mana_hwc_send_request(), mana_hwc_get_msg_index() is called: mana_hwc_send_request() -> mana_hwc_get_msg_index() -> down(&hwc->sema); Because hwc->sema is initialized to 1 (HW_CHANNEL_VF_BOOTSTRAP_QUEUE_DEPTH), only one HWC command can be in flight at a time. While the device is active, periodic work such as mana_gf_stats_work_handler() (running every 2 seconds without holding rtnl_lock) or auxiliary RDMA (mana_ib) callers can enter mana_hwc_send_request(), acquire hwc->sema, and sleep in wait_for_completion_timeout(&ctx->comp_event, ...) for the full 60s hwc_timeout when the hardware stops responding. When mana_dealloc_queues() resets the device and sets hwc->hwc_dead = true, it does not complete any in-flight ctx->comp_event, so the subsequent mana_disable_vport_rx() -> mana_hwc_send_request() call in mana_dealloc_queues() will still block in down(&hwc->sema) while holding rtnl_lock until the concurrent caller's timeout expires. Should hwc_timeout == 0 || hwc_dead also be checked before acquiring hwc->sema (or should in-flight waiters be completed when marking HWC dead)? > diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c > b/drivers/net/ethernet/microsoft/mana/mana_en.c > index 6b7c027e497ed..00885d21a9af0 100644 > --- a/drivers/net/ethernet/microsoft/mana/mana_en.c > +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c > @@ -3753,16 +3753,19 @@ int mana_attach(struct net_device *ndev) > static int mana_dealloc_queues(struct net_device *ndev) > { > struct mana_port_context *apc = netdev_priv(ndev); > - unsigned long timeout = jiffies + 120 * HZ; > struct gdma_dev *gd = apc->ac->gdma_dev; > + struct hw_channel_context *hwc; > + bool trigger_flr = false; > + int flr_pending, i, err; > struct mana_txq *txq; > struct sk_buff *skb; > - int i, err; > - u32 tsleep; > + u32 flr_txq_id = 0; > > if (apc->port_is_up) > return -EINVAL; > > + hwc = gd->gdma_context->hwc.driver_data; > + > if (apc->rxqs) > mana_chn_setxdp(apc, NULL); > > @@ -3774,42 +3777,92 @@ static int mana_dealloc_queues(struct net_device > *ndev) > * a txq because it may not timely see apc->port_is_up being cleared > * to false, but it doesn't matter since mana_start_xmit() drops any > * new packets due to apc->port_is_up being false. > - * > - * Drain all the in-flight TX packets. > - * A timeout of 120 seconds for all the queues is used. > - * This will break the while loop when h/w is not responding. > - * This value of 120 has been decided here considering max > - * number of queues. > + * Drain the in-flight TX completions for each queue. TX is already > + * stopped, so pending_sends can no longer grow. For each queue, > + * disable NAPI so it cannot run at the same time as this drain, > + * then process all completions still in the CQ. If some are still > + * missing after that, the hardware is not responding: reset device > + * and stop waiting on the now-unavailable HWC. > */ > > if (apc->tx_qp) { > for (i = 0; i < apc->num_queues; i++) { > + struct mana_cq *tx_cq; > + struct napi_struct *napi; > + int pending; > + > + if (!apc->tx_qp[i]) > + continue; > + > + tx_cq = &apc->tx_qp[i]->tx_cq; > txq = &apc->tx_qp[i]->txq; > - tsleep = 1000; > - while (atomic_read(&txq->pending_sends) > 0 && > - time_before(jiffies, timeout)) { > - usleep_range(tsleep, tsleep + 1000); > - tsleep <<= 1; > + napi = &tx_cq->napi; > + > + if (txq->napi_initialized) { > + /* Stop the NAPI poll so it cannot run > + * concurrently with the synchronous drain. > + */ > + napi_synchronize(napi); > + napi_disable_locked(napi); > + cancel_work_sync(&tx_cq->dim.work); > + > + /* Reap all TX completions currently in the CQ. > + * A zero budget frees skbs via the non-NAPI > + * path, which is required outside softirq. > + */ > + tx_cq->budget = 0; > + do { > + tx_cq->work_done = 0; > + mana_poll_tx_cq(tx_cq); > + } while (tx_cq->work_done); > } > - if (atomic_read(&txq->pending_sends)) { > - err = mana_gd_dev_reset(gd->gdma_context); > - if (err) { > - netdev_err(ndev, "device reset failed > %d with %d pkts pending in txq %u\n", > - err, > - atomic_read(&txq->pending_sends), > - txq->gdma_txq_id); > - } > - break; > + > + pending = atomic_read(&txq->pending_sends); > + if (!trigger_flr && pending > 0) { > + trigger_flr = true; > + flr_txq_id = txq->gdma_txq_id; > + flr_pending = pending; > } > } > > + if (trigger_flr && !(hwc && hwc->hwc_dead)) { > + /* Completions never arrived: the device is > + * unresponsive. Issue device reset and > + * mark the HWC dead so subsequent HWC commands > + * fail fast instead of each waiting the full > + * hwc_timeout. > + */ > + err = mana_gd_dev_reset(gd->gdma_context); > + if (err) > + netdev_err(ndev, "device reset failed %d with > %d pkts pending in txq %u\n", > + err, > + flr_pending, > + flr_txq_id); > + else if (hwc) > + WRITE_ONCE(hwc->hwc_dead, true); > + } [Severity: High] Won't this non-waiting CQ poll trigger a premature Function Level Reset and permanently mark the HWC dead whenever TX packets are still in flight in the hardware during a normal detach or per-port queue reset? In mana_detach(), setting apc->port_is_up = false and calling netif_tx_disable(ndev) prevents new packets from being queued in software, but packets posted to txq->gdma_sq right before netif_tx_disable() are still asynchronously in flight in the NIC hardware (DMA read, shaping, wire transmission, and CQE DMA write-back). mana_poll_tx_cq() is non-blocking: mana_poll_tx_cq() comp_read = mana_gd_poll_cq(cq->gdma_cq, completions, ...); if (comp_read < 1) return; As soon as mana_gd_poll_cq() reaches a CQE slot that the hardware has not yet written back to host memory, it returns 0, mana_poll_tx_cq() returns early with tx_cq->work_done == 0, and the do-while loop exits immediately without waiting for the in-flight SQ entries to complete. If any packet is still in flight on the NIC when the loop exits, atomic_read(&txq->pending_sends) > 0 sets trigger_flr = true, which calls mana_gd_dev_reset() and sets hwc->hwc_dead = true on healthy hardware. Because hwc->hwc_dead is never cleared without destroying and recreating the HWC channel, any operation that detaches and reattaches queues under traffic (such as mana_close() followed by mana_open(), mana_change_mtu(), mana_set_channels(), mana_set_ringparam(), mana_set_priv_flags(), or mana_xdp_set()) will cause subsequent mana_attach() or mana_open() calls and any other vPorts or RDMA instances on the function to fail permanently with -ETIMEDOUT. Similarly, single-port TX timeout recovery in mana_per_port_queue_reset_work_handler() runs when a TX queue is stalled (so pending_sends > 0 holds by definition): mana_per_port_queue_reset_work_handler() -> mana_detach(ndev, false) -> mana_dealloc_queues(ndev) -> mana_attach(ndev) Because mana_dealloc_queues() now resets the entire PCIe function and sets hwc->hwc_dead = true, the recovery call to mana_attach(ndev) immediately fails with -ETIMEDOUT instead of recovering the port. Furthermore, if mana_gd_dev_reset() fails or is skipped because hwc_dead was already set, the cleanup loop immediately below unmaps and frees the in-flight TX SKBs via mana_unmap_skb() and dev_kfree_skb_any() without having waited for DMA completion or reset the device. -- Sashiko AI review ยท https://sashiko.dev/#/patchset/[email protected]?part=1
