> -----Original Message----- > From: Intel-wired-lan <[email protected]> On Behalf Of Ivan > Vecera > Sent: Saturday, March 16, 2024 5:08 PM > To: [email protected] > Cc: Loktionov, Aleksandr <[email protected]>; moderated > list:INTEL ETHERNET DRIVERS <[email protected]>; open list > <[email protected]>; Chmielewski, Pawel > <[email protected]>; Eric Dumazet <[email protected]>; Nguyen, > Anthony L <[email protected]>; Hugo Ferreira <[email protected]>; > Keller, Jacob E <[email protected]>; Jakub Kicinski <[email protected]>; > Paolo Abeni <[email protected]>; David S. Miller <[email protected]> > Subject: [Intel-wired-lan] [PATCH net v4] i40e: Enforce software interrupt > during busy-poll exit > > As for ice bug fixed by commit b7306b42beaf ("ice: manage interrupts during > poll exit") followed by commit 23be7075b318 ("ice: fix software generating > extra interrupts") I'm seeing the similar issue also with i40e driver. > > In certain situation when busy-loop is enabled together with adaptive > coalescing, the driver occasionally misses that there are outstanding > descriptors to clean when exiting busy poll. > > Try to catch the remaining work by triggering a software interrupt when > exiting busy poll. No extra interrupts will be generated when busy polling is > not used. > > The issue was found when running sockperf ping-pong tcp test with adaptive > coalescing and busy poll enabled (50 as value busy_pool and busy_read sysctl > knobs) and results in huge latency spikes with more than 100000us. > > The fix is inspired from the ice driver and do the following: > 1) During napi poll exit in case of busy-poll (napo_complete_done() > returns false) this is recorded to q_vector that we were in busy > loop. > 2) Extends i40e_buildreg_itr() to be able to add an enforced software > interrupt into built value > 2) In i40e_update_enable_itr() enforces a software interrupt trigger > if we are exiting busy poll to catch any pending clean-ups > 3) Reuses unused 3rd ITR (interrupt throttle) index and set it to > 20K interrupts per second to limit the number of these sw interrupts. > > Test results > ============ > Prior: > [root@dell-per640-07 net]# sockperf ping-pong -i 10.9.9.1 --tcp -m 1000 > --mps=max -t 120 > sockperf: == version #3.10-no.git == > sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s) > > [ 0] IP = 10.9.9.1 PORT = 11111 # TCP > sockperf: Warmup stage (sending a few dummy messages)... > sockperf: Starting test... > sockperf: Test end (interrupted by timer) > sockperf: Test ended > sockperf: [Total Run] RunTime=119.999 sec; Warm up time=400 msec; > SentMessages=2438563; ReceivedMessages=2438562 > sockperf: ========= Printing statistics for Server No: 0 > sockperf: [Valid Duration] RunTime=119.549 sec; SentMessages=2429473; > ReceivedMessages=2429473 > sockperf: ====> avg-latency=24.571 (std-dev=93.297, mean-ad=4.904, > median-ad=1.510, siqr=1.063, cv=3.797, std-error=0.060, 99.0% ci=[24.417, > 24.725]) > sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order > messages = 0 > sockperf: Summary: Latency is 24.571 usec > sockperf: Total 2429473 observations; each percentile contains 24294.73 > observations > sockperf: ---> <MAX> observation = 103294.331 > sockperf: ---> percentile 99.999 = 45.633 > sockperf: ---> percentile 99.990 = 37.013 > sockperf: ---> percentile 99.900 = 35.910 > sockperf: ---> percentile 99.000 = 33.390 > sockperf: ---> percentile 90.000 = 28.626 > sockperf: ---> percentile 75.000 = 27.741 > sockperf: ---> percentile 50.000 = 26.743 > sockperf: ---> percentile 25.000 = 25.614 > sockperf: ---> <MIN> observation = 12.220 > > After: > [root@dell-per640-07 net]# sockperf ping-pong -i 10.9.9.1 --tcp -m 1000 > --mps=max -t 120 > sockperf: == version #3.10-no.git == > sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s) > > [ 0] IP = 10.9.9.1 PORT = 11111 # TCP > sockperf: Warmup stage (sending a few dummy messages)... > sockperf: Starting test... > sockperf: Test end (interrupted by timer) > sockperf: Test ended > sockperf: [Total Run] RunTime=119.999 sec; Warm up time=400 msec; > SentMessages=2400055; ReceivedMessages=2400054 > sockperf: ========= Printing statistics for Server No: 0 > sockperf: [Valid Duration] RunTime=119.549 sec; SentMessages=2391186; > ReceivedMessages=2391186 > sockperf: ====> avg-latency=24.965 (std-dev=5.934, mean-ad=4.642, > median-ad=1.485, siqr=1.067, cv=0.238, std-error=0.004, 99.0% ci=[24.955, > 24.975]) > sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order > messages = 0 > sockperf: Summary: Latency is 24.965 usec > sockperf: Total 2391186 observations; each percentile contains 23911.86 > observations > sockperf: ---> <MAX> observation = 195.841 > sockperf: ---> percentile 99.999 = 45.026 > sockperf: ---> percentile 99.990 = 39.009 > sockperf: ---> percentile 99.900 = 35.922 > sockperf: ---> percentile 99.000 = 33.482 > sockperf: ---> percentile 90.000 = 28.902 > sockperf: ---> percentile 75.000 = 27.821 > sockperf: ---> percentile 50.000 = 26.860 > sockperf: ---> percentile 25.000 = 25.685 > sockperf: ---> <MIN> observation = 12.277 > > Fixes: 0bcd952feec7 ("ethernet/intel: consolidate NAPI and NAPI exit") > Reported-by: Hugo Ferreira <[email protected]> > Reviewed-by: Michal Schmidt <[email protected]> > Signed-off-by: Ivan Vecera <[email protected]> > --- > Changes since v3 > - fixed kdoc warnings (thx Tony) > Changes since v2 > - eliminated two writes in hot-path (thx Jesse) Changes since v1 > - added Fixes: tag > > drivers/net/ethernet/intel/i40e/i40e.h | 1 + > drivers/net/ethernet/intel/i40e/i40e_main.c | 6 ++ > .../net/ethernet/intel/i40e/i40e_register.h | 3 + > drivers/net/ethernet/intel/i40e/i40e_txrx.c | 82 ++++++++++++++----- > drivers/net/ethernet/intel/i40e/i40e_txrx.h | 1 + > 5 files changed, 72 insertions(+), 21 deletions(-) >
Tested-by: Pucha Himasekhar Reddy <[email protected]> (A Contingent worker at Intel)
