Hi Jesse,
Jesse Brandeburg wrote:
> On Mon, 2009-12-21 at 18:53 -0700, Richard Scobie wrote:
>
>>I have a low end server, Core 2 Duo 2.8, 4GB used to backup using rsync
>>over a 82574L interface. Kernel 2.6.30.9-102.fc11.x86_64 (e1000e
>>0.3.3.4-k4). It is using MSI-X interrupts.
>>
>>It's suffering somewhat due to dropping frames:
>>
>>RX packets:294914332 errors:0 dropped:95203 overruns:0 frame:0
>>TX packets:355842341 errors:0 dropped:0 overruns:0 carrier:0
>>
>>and ethtool shows rx_missed_errors: 95203.
>>
>>Googling shows these are caused by the RX FIFO filling up.
>
>
> Hi Richard, can you give the whole ethtool -S output? depending on the
> value of rx_no_buffer_count, you may be able to do something.
rx_packets: 325509977
tx_packets: 360014787
rx_bytes: 441158607444
tx_bytes: 424925363743
rx_broadcast: 181969
tx_broadcast: 954
rx_multicast: 5725
tx_multicast: 7
rx_errors: 0
tx_errors: 0
tx_dropped: 0
multicast: 5725
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 103969
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 22181224
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 17013029
tx_flow_control_xoff: 17012765
rx_long_byte_count: 441158607444
rx_csum_offload_good: 325459865
rx_csum_offload_errors: 0
rx_header_split: 0
alloc_rx_buff_failed: 0
tx_smbus: 0
rx_smbus: 0
dropped_smbus: 0
rx_dma_failed: 0
tx_dma_failed: 0
> The other thing to send is the output of lspci -vvv for your system, I'm
> curious if ASPM is enabled for the ethernet port or its upstream port.
05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
Subsystem: Intel Corporation Gigabit CT Desktop Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at fcfe0000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at fcf00000 (32-bit, non-prefetchable) [size=512K]
Region 2: I/O ports at ec00 [size=32]
Region 3: Memory at fcfdc000 (32-bit, non-prefetchable) [size=16K]
Expansion ROM at fcf80000 [disabled] [size=256K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Mask- 64bit+ Count=1/1 Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1,
Latency L0 <128ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Capabilities: [a0] MSI-X: Enable+ Mask- TabSize=5
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [100] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn-
ChkCap- ChkEn-
Capabilities: [140] Device Serial Number f3-00-4c-ff-ff-21-1b-00
Kernel driver in use: e1000e
Kernel modules: e1000e
> The other thing we may be able to do is provide a patch to enable GRO if
> at all possible (which should help significantly if it is not already
> enabled,) you can check with ethtool -k ethX, but I guess it may already
> be on.
No, it appears to be off by default:
Offload parameters for eth0:
Cannot get device flags: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off
> Is flow control enabled to your switch? Are you using jumbo frames?
As you can see from the ethtool -S output above there are a large number
of tx_flow_control actuations.
Initially flow control was disabled and when I noticed the amount of
dropped packets, I enabled it and that stopped further packet loss.
However given the amount of flow control activity for the relatively
short time I enabled it, throughput is probably suffering, so I'd prefer
to leave it off, as this workload should be ethernet constrained as it
is feeding a large, fast RAID.
> There was a fifo (flow control) configuration issue in several versions
> of the e1000e driver in the kernel. If that was the case disabling flow
> control might help you, ethtool -A ethX autoneg off rx off tx off
>
> ethtool -G ethX rx 4096 will max out the number of rx descriptors.
>
> you also may benefit from decreasing the interrupt rate using
> ethtool -C ethX rx-usecs 125 (8000 interrupts per second) because you're
> not doing a latency sensitive workload
I'll give it a try.
> Please also provide /proc/interrupts and ethtool -e ethX, and if you are
CPU0 CPU1
0: 226 481 IO-APIC-edge timer
1: 0 2 IO-APIC-edge i8042
8: 26 26 IO-APIC-edge rtc0
9: 0 0 IO-APIC-fasteoi acpi
12: 3 1 IO-APIC-edge i8042
16: 0 0 IO-APIC-fasteoi pata_marvell
17: 1694 410198 IO-APIC-fasteoi sata_sil24
28: 46793552 22812 PCI-MSI-edge ahci
29: 44433942 6 PCI-MSI-edge eth0-rx-0
30: 15 58772791 PCI-MSI-edge eth0-tx-0
31: 97668 3 PCI-MSI-edge eth0
32: 48 48 PCI-MSI-edge ioc0
NMI: 0 0 Non-maskable interrupts
LOC: 27702671 26695738 Local timer interrupts
SPU: 0 0 Spurious interrupts
RES: 1176280 2826251 Rescheduling interrupts
CAL: 265 516 Function call interrupts
TLB: 4901 1653 TLB shootdowns
TRM: 0 0 Thermal event interrupts
THR: 0 0 Threshold APIC interrupts
ERR: 0
MIS: 0
0x0000 00 1b 21 4c 00 f3 20 04 46 f7 80 10 ff ff ff ff
0x0010 69 e4 03 81 6b 02 1f a0 86 80 d3 10 ff ff 58 9c
0x0020 00 00 01 20 94 7e ff ff 00 10 48 00 00 00 04 27
0x0030 c9 6c 50 31 3e 07 0b 46 84 2d 40 01 00 f0 06 07
0x0040 00 60 00 71 08 14 ff ff 01 4d ec 92 5c fc 83 f0
0x0050 28 00 33 02 50 00 1f 7d 61 19 53 04 a0 00 ff ff
0x0060 00 01 00 40 15 13 03 40 ff ff ff ff ff ff ff ff
0x0070 ff ff ff ff ff ff ff ff ff ff 30 01 ff ff d5 52
0x0080 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0090 ff ff ff ff 00 c7 ff ff ff ff ff ff ff ff ff ff
0x00a0 00 00 1f 38 0a 00 04 00 80 02 00 20 ff ff ff 7f
0x00b0 00 00 1f 38 08 00 04 00 00 00 1f 38 09 00 04 00
0x00c0 00 00 3f 0a 14 00 20 00 01 7e 5f 10 3f 00 20 00
0x00d0 00 7c 9f 61 00 08 00 00 00 7c 5f 62 00 00 30 00
0x00e0 01 04 e6 08 ca fa e3 7f 02 07 00 20 02 04 06 09
0x00f0 03 04 26 09 01 7a e4 7f 81 03 00 20 0e 7a e4 7f
0x0100 02 00 1f 38 c4 04 04 00 0e 08 26 08 04 02 89 10
0x0110 10 04 26 08 06 02 49 10 01 12 49 10 00 00 1f 38
0x0120 30 03 04 00 00 00 1f 38 07 00 04 00 ff ff ff ff
0x0130 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0140 00 00 1f 38 0a 00 04 00 00 0a 00 20 ef ef ef ef
0x0150 00 00 1f 38 0b 00 04 00 00 00 1f 38 03 00 04 00
0x0160 00 00 1f 38 05 00 04 00 00 00 1f 38 02 00 04 00
0x0170 00 07 00 20 ef ef ef ef 00 04 00 00 04 04 00 00
0x0180 08 08 00 08 04 00 08 00 04 00 04 00 ff 00 00 00
0x0190 ff 00 00 04 00 00 00 00 00 00 00 00 00 00 df 09
0x01a0 98 11 00 00 00 00 1f 38 01 00 04 00 00 fd e4 67
0x01b0 00 01 00 00 82 12 00 20 00 fd e5 7f 81 50 00 00
0x01c0 82 0a 00 20 06 7a e2 7f 82 09 00 20 08 fe 04 40
0x01d0 00 fe 3f 60 02 02 80 11 02 02 80 11 02 02 80 11
0x01e0 02 7c 80 11 57 01 00 00 02 7c 40 11 06 00 00 a1
0x01f0 78 fe 6a 50 04 80 21 08 02 fe 5f 60 40 02 1f 38
0x0200 e0 04 04 00 00 80 21 08 04 fe 5f 60 40 02 1f 38
0x0210 e0 04 04 00 0f fe ff 61 17 7a e6 7f 02 02 00 20
0x0220 00 17 e5 47 02 01 00 20 24 fc 84 10 00 00 00 00
0x0230 00 00 df 09 98 11 00 00 00 17 e5 47 01 fc c6 79
0x0240 00 00 00 00 00 1c 1f 10 98 11 00 00 00 00 1f 38
0x0250 0c 00 04 00 ff ff ff ff ff ff ff ff ff ff ff ff
0x0260 69 53 84 03 01 00 00 00 00 00 00 00 00 00 00 00
0x0270 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0290 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0300 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0310 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0320 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0330 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0340 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0350 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0360 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0370 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0380 00 00 00 00 00 00 00 00 03 3d 00 00 00 00 00 00
0x0390 00 00 00 00 00 00 00 00 00 00 00 00 bc 0c 00 00
0x03a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x03b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x03c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x03d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x03e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x03f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0400 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0420 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0430 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0450 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0470 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0480 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0490 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0500 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0510 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0520 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0530 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0540 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0550 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0560 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0570 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0580 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0590 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05e0 00 00 00 00 ff ff ff ff ff ff ff ff ff ff ff ff
All values from this point are ff.
> feeling gung-ho, the output of the ethregs utility available at
> sourceforge (you'll have to build it) in the Register Dump utility
> section.
I have built this but the output is quite prodigious. Does the list
accept zipped attachments?
Regards,
Richard
------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
