** Description changed: SRU Justification: [Impact] - On BlueField systems using the mlxbf_gige driver (the OOB management - Ethernet interface), the RX path can stall permanently. If receive - completions (CQEs) arrive after NAPI's final check but before the RX - interrupt is unmasked, and they fill the RX ring, the hardware has - nowhere to write further completions. The interrupt only fires when a - CQE is written while it is unmasked, so it never fires again. NAPI is - never rescheduled and the interface stops receiving packets. + The mlxbf_gige receive path can leave completed packets pending when + packets arrive between the last completion check and interrupt unmasking. + If the receive ring fills and no new interrupt is generated, RX can stall, + interrupting BlueField out-of-band management connectivity. + + This submission targets linux-bluefield in Ubuntu 24.04 LTS (Noble), + using the 6.8 kernel. The reproducer uses SF creation, activation, and + unbind/bind activity while monitoring out-of-band ICMP and SSH + reachability. [Fix] - * After mlxbf_gige_poll() completes NAPI and unmasks the RX interrupt, - it now checks whether CQEs are already pending (new helper - mlxbf_gige_rx_pending(), comparing RX_CQE_PACKET_CI against RX_WQE_PI). - If so, it calls napi_schedule() to process them. This closes the window - where CQEs written while the interrupt was masked fill the ring, so no - new interrupt is ever raised and RX stalls. + After completing NAPI and unmasking receive interrupts, check whether + hardware has posted completions not yet consumed by software. Reschedule + NAPI when pending completions exist. The index comparison accounts for + initial receive-buffer credits, 16-bit counter wrap, and a full ring. - [Test Case] + [Test Plan] - * Sustained throughput test cases and scaling use cases with large - number of SF interfaces and CPU utilization + 1. Build the patched linux-bluefield kernel/module for the affected + hardware. Record the kernel version and driver build ID. + 2. Establish an unpatched baseline using the same SF stress workload and + RX traffic. Capture ring indices, interrupt state, and OOB connectivity + when RX stops making progress. + 3. Repeat with the patch and no diagnostic watchdog for at least the + baseline failure interval. Record duration, traffic, SF count, and any + ICMP/SSH loss. Verify that RX continues making progress. + 4. Exercise sustained RX across repeated NAPI budget exhaustion, index + wrap, and full-ring conditions. Compare packet drops and throughput. + 5. Cycle the interface down/up and unload/reload the driver where supported + while traffic is active. Check for warnings, hangs, and teardown races. + 6. Repeat validation with the Noble linux-bluefield kernel in -proposed + when available, and report the result on the Launchpad bug. + [Where problems could occur] - [Regression Potential] + Additional register reads and NAPI rescheduling affect RX completion and + interrupt rearming. Incorrect assumptions about counter width, initial + credits, or wrap could miss pending packets or cause unnecessary polling. + Review interrupt masking after software-triggered rescheduling and races + with another poll, interface shutdown, and driver removal after + napi_complete_done() releases NAPI ownership. Check throughput, CPU use, + and interrupt rates under sustained RX load. - Adds additional register read overhead but should not cause functional - regression + [Other Info] + + This is a locally developed change. Upstream submission is intended; + there is no accepted upstream commit or posted upstream thread to cite. + SAUCE inclusion is requested to address the Noble OOB RX stall without + waiting for mainline integration. After upstream integration, cite that + commit for later backports.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2170231 Title: mlxbf_gige: RX hang after NAPI poll unmasks interrupt with CQEs already pending To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-bluefield/+bug/2170231/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
