On Fri, Nov 13, 2026 at 02:04:08PM +0530, Sahil Chandna wrote: > On 12-11-2026 17:37, Breno Leitao wrote: > > Hello Sahil, > > > > On Thu, Nov 12, 2026 at 04:57:18AM -0600, Sahil Chandna wrote: > >> Implement a graduated recovery approach: on the first TX timeout (or > >> when the previous timeout was more than 2 minutes ago), schedule NAPI on > >> the stalled queue's TX CQ via napi_schedule(). This triggers > >> mana_poll_tx_cq() through the normal NAPI path, processing any pending > >> completions without relying on the interrupt and recovering > >> missed-interrupt scenarios without queue reset. > > > > Which interrupt is being missed here? > > > Hi Breno, > > It is the TX CQ completion interrupt. After NAPI finishes, it re-arms > the TX CQ. When a new CQE arrives, the hardware post EQE, which raises > the MSI-X and schedules NAPI. If that notification for the TX CQ is lost > , NAPI never runs for the TX CQ and pending_sends never drains and the > queue stays stopped, so napi_schedule() polls the CQ, reaps the CQEs > and re-arms it.
Thanks. Which one gets lost, the EQE or the MSI-X? > >> If any subsequent TX timeout fires on any queue, within 2 minutes of > >> the previous one, escalate to the existing port-level detach/attach > >> reset, as the CQE poll did not recover the queue. > > > > How was 2 minutes chosen? mana sets watchdog_timeo to 15 * HZ, no? > The watchdog_timeo is TX stall timeout setting i.e if a queue remains > stalled for this much time, it trigger watchdog timeout. Here, 2 minute > is not timeout/wait. The 2 minutes here is for deciding whether a new > queue stall event is a repeat of the previous incident or a new, > unrelated one that should get the cheap poll first. For e.g. if a queue > gets stalled, 15 * HZ sec later, timeout occurs and this patch trigger > manual NAPI, if another stall occurs (it can be on any queue) within 2 > minutes of last one, the existing recovery mechanism of mana > detach/attach will be tried.The patch is adding a first level less > destructive recovery first. > For selecting 2 minutes, I do not have any hardware basis for it afaik, > but I can confirm and check with our host team as well. So the goal is to tell an isolated stall from a wider problem. A port-wide timestamp only approximates that. It cannot tell the same queue from another one, and since dev_watchdog() reports one queue per run, two queues that stall together show up ~15s apart: the first gets a kick, the second gets the port reset anyway, no? Could mana_tx_timeout() check this directly? Look at the other TX queues the way dev_watchdog() does (stopped, with trans_start older than watchdog_timeo). If more than one is stalled, reset right away. Otherwise kick this queue, and reset if the same queue times out again within a couple of watchdog periods, as gve_tx_timeout_try_q_kick() does. That also gets rid of the 2 minutes.
