> -----Original Message-----
> From: Sahil Chandna <[email protected]>
> Sent: Wednesday, November 18, 2026 4:14 AM
> To: Haiyang Zhang <[email protected]>; [email protected]; Dexuan
> Cui <[email protected]>; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected]; Konstantin
> Taranov <[email protected]>; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; linux-
> [email protected]; [email protected]; linux-
> [email protected]; [email protected]
> Cc: [email protected]
> Subject: [PATCH net v4] net: mana: Avoid waiting on dead HWC after device
> reset
> 
> After device reset is issued due to TX drain timeout in
> mana_dealloc_queues(), hardware channel (HWC) state is non-functional.
> However, the driver continues to issue HWC commands
> (mana_disable_vport_rx,
> mana_fence_rqs, mana_destroy_wq_obj) which will each wait for a
> completion that never arrives.
> 
> The first such command (MANA_CONFIG_VPORT_RX) blocks for the full
> hwc_timeout (60s as set by the firmware on the tested system), after
> which the timeout is reduced to 1 ms for subsequent commands. This
> causes the calling thread to hold the RTNL lock for over 180 seconds
> (120s TX drain + 60s HWC timeout), triggering hung task warnings and
> blocking all other networking operations on the system.
> 
> Fix this by:
> 1. Replacing the sleep-based TX drain with a CQE polling approach:
>    disable NAPI per queue and poll its CQ synchronously, with up to 3
>    more polls 100 ms apart to collect late completions. Reset the
>    device only if sends remain unacknowledged after that, then free
>    uncompleted SKBs. The device is reset at most once per call, and
>    not at all if an earlier reset already marked the HWC dead. Once a
>    queue is found stuck or the HWC is dead, the remaining queues are
>    polled only once. This replaces the 120s drain timeout with at most
>    300 ms of sleep per queue. If queue is still stuck,
>    mana_gd_dev_reset() waits for device to return. The 300 ms budget
>    is not based on a hardware limit but in testing under load, the
>    completions still outstanding on a queue had all arrived by its
>    second poll.
> 2. Setting the hwc_dead flag after a successful device reset. If the
>    reset fails, the device may not have been reset and may still access
>    memory, so teardown commands are still posted.
> 3. Checking the hwc_dead flag in mana_hwc_send_request() before taking
>    an HWC message slot and again before posting the WQE, and returning
>    -ETIMEDOUT immediately without submitting commands to hardware that
>    cannot respond. A request that was already posted, or that passed
>    these checks just before hwc_dead was set (for example from the
>    gf_stats worker or mana_ib, which do not hold RTNL), still waits for
>    its own timeout, as before this patch.
> 4. Skipping mana_fence_rqs() and the SHM channel teardown handshake
>    once hwc_dead is set.
> 
> Fixes: a7dfeda6fdec ("net: mana: Fix MANA VF unload when hardware is
> unresponsive")
> Cc: [email protected]
> Co-developed-by: Dipayaan Roy <[email protected]>
> Signed-off-by: Dipayaan Roy <[email protected]>
> Signed-off-by: Sahil Chandna <[email protected]>
> ---

Reviewed-by: Haiyang Zhang <[email protected]>



Reply via email to