Module Name: src Committed By: msaitoh Date: Tue Feb 20 07:24:37 UTC 2018
Modified Files: src/sys/dev/pci/ixgbe: ixgbe.c ixgbe_type.h Log Message: - Fix a bug that RX may stall on heavy load on ixg(4) derived from FreeBSD's AIM (Auto Interrupt Moderation) bug. When I use a machine as a NFS client, sometimes one of queue pairs doesn't get any interrupt other than every second tick via ixgbe_local_timer1(). When the problem occured, the queue pair's hw.ixgM.qN.interrupt_rate is always 500000. When this problem occuring, set hw.ixgM.qN.interrupt_rate lower than 166667 recover from stall. i.e.: sysctl -w hw.ixgM.qN.interrupt_rate=166667 (don't revocer) sysctl -w hw.ixgM.qN.interrupt_rate=166666 (recover) Relatios between the interrupt_rate and EICR's ITR_INTERVAL field is as follows: int_rate | EICR[11:0] | interval in us | recover | |(ITR_INTERVAL)| (10G and 1G) | | ---------+--------------+----------------+---------+ 500000 | 0x008(0) | 2 | not | 166667 | 0x010(1) | 4 | not | 166666 | 0x018(2) | 6 | recover | The reason why int_rate becomes 500000 is that xgbe_tx_eof() doesn't increment rxr->packets(*1). Even if we fix rxr->packets' bug, interrupt_rate might become greater than 166666 and it might cause stall. While reading datasheets, knakahara noticed a section titled with "ITR Affect on RSC Functionality". It says "When RSC is enabled on specific RX queues, the associated ITR interval with these queus must be enabled and must be larger (in time uints) than RSC delay". Currently, RSC_DELAY field in the GPIE register is 0 and it means 4us for 10G and 1G. The greater ITR_INTERVAL value of 4us is 6us == 166666. Yes, BINGO! This description is noted in 82599 and newer datasheets and not in 82598 datasheet. I don't know if 82598 has this limitation but, I apply this limitation all of chips. (*1) Note that this bug is going to be fixed in the next commit to distinct between two different bugs. - The bitfield of EITR register is different between 82598 and others. Only ixgbe_msix_que() taken care of it. Make new function ixgbe_eitr_write() and use it in all of functions which modify ITR_INTERVAL. XXX pullup-8 To generate a diff of this commit: cvs rdiff -u -r1.123 -r1.124 src/sys/dev/pci/ixgbe/ixgbe.c cvs rdiff -u -r1.30 -r1.31 src/sys/dev/pci/ixgbe/ixgbe_type.h Please note that diffs are not public domain; they are subject to the copyright notices on the relevant files.