Thank you for your contribution! Sashiko AI review found 2 potential issue(s) 
to consider:
- [High] net: mana: missing TX CQ poll after final sleep in 
mana_dealloc_queues()
- [Medium] net: mana: incomplete dead HWC checks in mana_dealloc_queues()
--

commit 4704079fb73cc77fdb1cc7581dd8e498b290c328
Author: Sahil Chandna <[email protected]>

net: mana: Avoid waiting on dead HWC after device reset

This commit replaces the 120s sleep-based TX drain in mana_dealloc_queues()
with synchronous CQ polling and marks the hardware channel dead after a
device reset so subsequent HWC commands fail fast instead of waiting for a
timeout.

> diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c 
> b/drivers/net/ethernet/microsoft/mana/mana_en.c
> index 6b7c027e497ed..2bf17fdcfa2bd 100644
> --- a/drivers/net/ethernet/microsoft/mana/mana_en.c
> +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c

[ ... ]

> @@ -3774,42 +3784,104 @@ static int mana_dealloc_queues(struct net_device 
> *ndev)
>        * a txq because it may not timely see apc->port_is_up being cleared
>        * to false, but it doesn't matter since mana_start_xmit() drops any
>        * new packets due to apc->port_is_up being false.
> -      *
> -      * Drain all the in-flight TX packets.
> -      * A timeout of 120 seconds for all the queues is used.
> -      * This will break the while loop when h/w is not responding.
> -      * This value of 120 has been decided here considering max
> -      * number of queues.
> +      * Drain the in-flight TX completions for each queue. TX is already
> +      * stopped, so pending_sends can no longer grow. For each queue,
> +      * disable NAPI so it cannot run at the same time as this drain,
> +      * then process all completions still in the CQ. If some are still
> +      * missing after that, the hardware is not responding: reset device
> +      * and stop waiting on the now-unavailable HWC.
>        */
>  
>       if (apc->tx_qp) {
>               for (i = 0; i < apc->num_queues; i++) {
> +                     struct napi_struct *napi;
> +                     struct mana_cq *tx_cq;
> +                     int pending;
> +
> +                     if (!apc->tx_qp[i])
> +                             continue;
> +
> +                     tx_cq = &apc->tx_qp[i]->tx_cq;
>                       txq = &apc->tx_qp[i]->txq;
> -                     tsleep = 1000;
> -                     while (atomic_read(&txq->pending_sends) > 0 &&
> -                            time_before(jiffies, timeout)) {
> -                             usleep_range(tsleep, tsleep + 1000);
> -                             tsleep <<= 1;
> -                     }
> -                     if (atomic_read(&txq->pending_sends)) {
> -                             err = mana_gd_dev_reset(gd->gdma_context);
> -                             if (err) {
> -                                     netdev_err(ndev, "device reset failed 
> %d with %d pkts pending in txq %u\n",
> -                                                err,
> -                                         atomic_read(&txq->pending_sends),
> -                                         txq->gdma_txq_id);
> +                     napi = &tx_cq->napi;
> +
> +                     if (txq->napi_initialized) {
> +                             /* Stop the NAPI poll so it cannot run
> +                              * concurrently with the synchronous drain.
> +                              */
> +                             napi_synchronize(napi);
> +                             napi_disable_locked(napi);
> +                             cancel_work_sync(&tx_cq->dim.work);
> +
> +                             /* Reap all TX completions currently in the CQ.
> +                              * For completion which have still not arrived
> +                              * allow time to settle.
> +                              * A zero budget frees skbs via the non-NAPI
> +                              * path, which is required outside softirq.
> +                              */
> +                             tx_cq->budget = 0;
> +                             for (retry = 0; retry < MANA_TX_DRAIN_RETRIES;
> +                                  retry++) {
> +                                     do {
> +                                             tx_cq->work_done = 0;
> +                                             mana_poll_tx_cq(tx_cq);
> +                                     } while (tx_cq->work_done);
> +
> +                                     if (!atomic_read(&txq->pending_sends) ||
> +                                         trigger_flr)
> +                                             break;
> +
> +                                     msleep(100);
>                               }
> -                             break;
>                       }
> +
> +                     pending = atomic_read(&txq->pending_sends);
> +
> +                     if (!trigger_flr && pending > 0) {
> +                             trigger_flr = true;
> +                             flr_txq_id = txq->gdma_txq_id;
> +                             flr_pending = pending;
> +                     }
> +             }

[Severity: High]
In mana_dealloc_queues(), NAPI is disabled on tx_cq via napi_disable_locked()
before entering the MANA_TX_DRAIN_RETRIES loop, so only mana_poll_tx_cq()
inside this loop can reap CQEs and decrement txq->pending_sends.

On the last iteration (retry == MANA_TX_DRAIN_RETRIES - 1), if
txq->pending_sends is still non-zero after polling, the loop calls
msleep(100) and then exits without calling mana_poll_tx_cq(tx_cq) again.

Because NAPI is disabled, any TX completions that arrive during that final
100 ms sleep remain unpolled in the CQ when pending =
atomic_read(&txq->pending_sends) is checked right after the loop. Can this
cause mana_dealloc_queues() to miss completions that arrived during the final
sleep and trigger an unnecessary mana_gd_dev_reset() and hwc->hwc_dead = true?

> +
> +             if (trigger_flr && !(hwc && hwc->hwc_dead)) {
> +                     /* Completions never arrived: the device is
> +                      * unresponsive. Issue device reset and
> +                      * mark the HWC dead so subsequent HWC commands
> +                      * fail fast instead of each waiting the full
> +                      * hwc_timeout.
> +                      */
> +                     err = mana_gd_dev_reset(gd->gdma_context);
> +                     if (err)
> +                             netdev_err(ndev, "device reset failed %d with 
> %d pkts pending in txq %u\n",
> +                                        err,
> +                                        flr_pending,
> +                                        flr_txq_id);
> +                     else if (hwc)
> +                             WRITE_ONCE(hwc->hwc_dead, true);
>               }
>  
>               for (i = 0; i < apc->num_queues; i++) {
> +                     struct napi_struct *napi;
> +                     struct mana_cq *tx_cq;
> +
> +                     if (!apc->tx_qp[i])
> +                             continue;
> +
> +                     tx_cq = &apc->tx_qp[i]->tx_cq;
> +                     napi = &tx_cq->napi;
> +
>                       txq = &apc->tx_qp[i]->txq;
>                       while ((skb = skb_dequeue(&txq->pending_skbs))) {
>                               mana_unmap_skb(skb, apc);
>                               dev_kfree_skb_any(skb);
>                       }
>                       atomic_set(&txq->pending_sends, 0);
> +
> +                     if (txq->napi_initialized) {
> +                             netif_napi_del_locked(napi);
> +                             txq->napi_initialized = false;
> +                     }
>               }
>       }
>  
> @@ -3822,7 +3894,8 @@ static int mana_dealloc_queues(struct net_device *ndev)
>       if (err && mana_en_need_log(apc, err))
>               netdev_err(ndev, "Failed to disable vPort: %d\n", err);
>  
> -     mana_fence_rqs(apc);
> +     if (!(hwc && hwc->hwc_dead))
> +             mana_fence_rqs(apc);

[Severity: Medium]
Should the dead-HWC checks in mana_dealloc_queues() also account for
hwc->hwc_timeout == 0 (or mana_disable_vport_rx() timing out) and check
whether the HWC is already dead inside the TX drain retry loop?

Specifically:

1. In mana_serv_reset(), an unresponsive HWC is marked by setting
   WRITE_ONCE(hwc->hwc_timeout, 0) before calling
   mana_gd_suspend() -> mana_remove() -> mana_detach() ->
   mana_dealloc_queues():

   drivers/net/ethernet/microsoft/mana/gdma_pci.c:mana_serv_reset() {
       ...
       WRITE_ONCE(hwc->hwc_timeout, 0);
       dev_info(&pdev->dev, "MANA reset cycle start\n");
       mana_gd_suspend(pdev, PMSG_SUSPEND);
   }

   If no TX packets are pending (so trigger_flr is false and hwc->hwc_dead
   stays false), or if the HWC dies while TX is idle and
   mana_disable_vport_rx() fails with -ETIMEDOUT (reducing hwc->hwc_timeout
   to 1 without setting hwc->hwc_dead), checking only !(hwc && hwc->hwc_dead)
   still calls mana_fence_rqs(apc), which logs an error and sleeps 100 ms per
   RX queue while holding rtnl_lock.

2. In the TX drain retry loop above, the early break only checks the local
   variable trigger_flr:

       if (!atomic_read(&txq->pending_sends) ||
           trigger_flr)
           break;

   On a multi-port device during mana_remove(), after the first port resets
   the device and sets hwc->hwc_dead = true (or when hwc->hwc_timeout == 0),
   subsequent calls to mana_dealloc_queues() start with trigger_flr == false
   and will still sleep through MANA_TX_DRAIN_RETRIES on the first queue with
   pending sends.

-- 
Sashiko AI review ยท 
https://sashiko.dev/#/patchset/[email protected]?part=1

Reply via email to