Re: [Aoetools-discuss] end_request: I/O error
Hi! Probably my MTU is too high? Thanks again. I actually don't think so. A MTU of 9000 is quite common and the Dell interface (em1) you showed us is quite capable. @Ed: Excellent hint to use aoe-sancheck! I completely missed that... # aoe-sancheck [...] eth2 UP 9000 14e4:164f [...] Device Macs Payload Local Interfaces e50.0 1 11776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload [[...] $ ethtool -k em1 Features for em1: [...] I just ask myself, why aoe-sancheck finds your aoe storage via eth2 (which should then be em3?) and not via em1. Did you change something on your setup? -- Adi -- ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] end_request: I/O error
Probably my MTU is too high? Thanks again. # aoe-sancheck Probing...skipping eth3, discover failure: Network is down skipping eth1, discover failure: Network is down done. == INTERFACE SUMMARY == NameStatus MTU PCI ID eth0UP 150014e4:163a eth1DN 150014e4:163a eth2UP 900014e4:164f eth3DN 150014e4:164f == DEVICE SUMMARY == Device MacsPayload Local Interfaces e50.0 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.1 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.2 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.3 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.4 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.5 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.6 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.7 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.8 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.9 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.10 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.11 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.12 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.13 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.14 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.15 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.16 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.17 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.18 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.19 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.20 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.21 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.22 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e50.23 111776 eth2 The path eth2-002590048ba3 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e52.0 115872 eth2 The path eth2-003048dea511 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e52.1 115872 eth2 The path eth2-003048dea511 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's capable payload e52.2 115872 eth2 The path eth2-003048dea511 is only capable of 1024 byte payloads eth2: MTU (9000) not set optimally for device's
Re: [Aoetools-discuss] end_request: I/O error
Hi Adi, Thanks for your reply. Honestly..I don't know what those dropped packets, it;s just a new installed system. Do you mean my MTU 9000 is too high? Daofeng On Tue, Jun 16, 2015 at 11:52 AM, Adi Kriegisch a...@cg.tuwien.ac.at wrote: Hey! I was trying to relocating my storage from 1 server to another, the old one has aoe driver version 81 installed, the new has 85 installed, I see this error message on the new system only. Is there a way to fix them? [...] Jun 15 17:11:01 cluster kernel: [ 1243.965424] end_request: I/O error, dev etherd/e50.23, sector 1953523728 [...] # aoe-stat e50.23 1000.204GBem1 8704 up [...] root@cluster:/home/d# ifconfig em1 Link encap:Ethernet HWaddr 5c:f9:dd:b9:48:a8 inet addr:10.200.0.20 Bcast:10.200.0.255 Mask:255.255.255.0 inet6 addr: fe80::5ef9:ddff:feb9:48a8/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:392548 errors:0 dropped:337 overruns:0 frame:0 TX packets:311526 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:164961826 (164.9 MB) TX bytes:20663984 (20.6 MB) Strange. Do you have any idea what the dropped packets (337 above) are? Other than that, are there any offloading features (ethtool) enabled on the nic that cannot reliably deal with jumbo frames? -- Adi -- ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] end_request: I/O error
Hey! Honestly..I don't know what those dropped packets, it;s just a new installed system. I see... depending on what services are running on that network, this might be some avahi messages or similar stuff; in case this is a storage-only network and there is no other stuff floating around, this may well be the cause for the issues... Do you mean my MTU 9000 is too high? No, I don't think so. Maybe some of the ethernet offloading stuff in the NIC is unable to handle larger packets? You may check with 'ethtool -k em1' what is enabled and may try to selectively disable and retest... -- Adi -- ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] end_request: I/O error
Some output from dmesg: [67285.437643] systemd-udevd[7632]: starting version 204 [68228.934696] aoe: AoE v85 initialised. [68228.935045] aoe: e52.23: setting 8704 byte data frames [68228.935073] aoe: e50.23: setting 8704 byte data frames [68228.935081] aoe: e52.22: setting 8704 byte data frames [68228.935098] aoe: e50.22: setting 8704 byte data frames [68228.935104] aoe: e52.21: setting 8704 byte data frames [68228.935110] aoe: e50.21: setting 8704 byte data frames [68228.935121] aoe: e52.20: setting 8704 byte data frames [68228.935133] aoe: e50.20: setting 8704 byte data frames [68228.935146] aoe: e52.19: setting 8704 byte data frames [68228.935152] aoe: e50.19: setting 8704 byte data frames [68228.935163] aoe: e52.18: setting 8704 byte data frames [68228.935169] aoe: e52.17: setting 8704 byte data frames [68228.935180] aoe: e50.18: setting 8704 byte data frames [68228.935188] aoe: e52.16: setting 8704 byte data frames [68228.935197] aoe: e50.17: setting 8704 byte data frames [68228.935203] aoe: e52.15: setting 8704 byte data frames [68228.935212] aoe: e50.16: setting 8704 byte data frames [68228.935220] aoe: e52.14: setting 8704 byte data frames [68228.935227] aoe: e50.15: setting 8704 byte data frames [68228.935233] aoe: e52.13: setting 8704 byte data frames [68228.935242] aoe: e50.14: setting 8704 byte data frames [68228.935250] aoe: e52.12: setting 8704 byte data frames [68228.935258] aoe: e50.13: setting 8704 byte data frames [68228.935265] aoe: e52.11: setting 8704 byte data frames [68228.935274] aoe: e50.12: setting 8704 byte data frames [68228.935281] aoe: e52.10: setting 8704 byte data frames [68228.935290] aoe: e50.11: setting 8704 byte data frames [68228.935297] aoe: e52.9: setting 8704 byte data frames [68228.935306] aoe: e52.8: setting 8704 byte data frames [68228.935316] aoe: e50.10: setting 8704 byte data frames [68228.935323] aoe: e52.7: setting 8704 byte data frames [68228.935330] aoe: e50.9: setting 8704 byte data frames [68228.935347] aoe: e52.6: setting 8704 byte data frames [68228.935356] aoe: e50.8: setting 8704 byte data frames [68228.935365] aoe: e52.5: setting 8704 byte data frames [68228.935373] aoe: e50.7: setting 8704 byte data frames [68228.935380] aoe: e52.4: setting 8704 byte data frames [68228.935389] aoe: e50.6: setting 8704 byte data frames [68228.935395] aoe: e52.3: setting 8704 byte data frames [68228.935404] aoe: e50.5: setting 8704 byte data frames [68228.935413] aoe: e52.2: setting 8704 byte data frames [68228.935423] aoe: e50.4: setting 8704 byte data frames [68228.935433] aoe: e52.1: setting 8704 byte data frames [68228.935443] aoe: e50.3: setting 8704 byte data frames [68228.935451] aoe: e52.0: setting 8704 byte data frames [68228.935458] aoe: e50.2: setting 8704 byte data frames [68228.935467] aoe: e50.1: setting 8704 byte data frames [68228.935475] aoe: e50.0: setting 8704 byte data frames [68228.935505] aoe: 00259005ba18 e50.23 vace0 has 1953525117 sectors [68228.935861] aoe: 00259005ba18 e50.22 vace0 has 1953525117 sectors [68228.936182] aoe: 00259005ba18 e50.21 vace0 has 1953525117 sectors [68228.936477] aoe: 00259005ba18 e50.20 vace0 has 1953525117 sectors [68228.936720] aoe: 00259005ba18 e50.19 vace0 has 1953525117 sectors [68228.937132] aoe: 00259005ba18 e50.18 vace0 has 1953525117 sectors [68228.937316] etherd/e50.23: unknown partition table [68228.937524] aoe: 00259005ba18 e50.17 vace0 has 1953525117 sectors [68228.937600] aoe: 00259005ba18 e50.16 vace0 has 1953525117 sectors [68228.937698] aoe: 00259005ba18 e50.15 vace0 has 1953525117 sectors [68228.937704] etherd/e50.22: unknown partition table [68228.938019] aoe: 00259005ba18 e50.14 vace0 has 1953525117 sectors [68228.938114] etherd/e50.21: unknown partition table [68228.938372] aoe: 00259005ba18 e50.13 vace0 has 1953525117 sectors [68228.938693] aoe: 00259005ba18 e50.12 vace0 has 1953525117 sectors [68228.938703] etherd/e50.20: unknown partition table [68228.938903] etherd/e50.19: unknown partition table [68228.939022] aoe: 00259005ba18 e50.11 vace0 has 1953525117 sectors [68228.939167] etherd/e50.18: unknown partition table [68228.939447] aoe: 00259005ba18 e50.10 vace0 has 1953525117 sectors [68228.939510] etherd/e50.17: unknown partition table [68228.939731] etherd/e50.16: unknown partition table [68228.939913] etherd/e50.15: unknown partition table [68228.940178] etherd/e50.14: unknown partition table [68228.940386] aoe: 003048dd4d24 e52.23 vace0 has 1953525117 sectors [68228.940394] aoe: 003048dd4d24 e52.22 vace0 has 1953525117 sectors [68228.940399] aoe: 003048dd4d24 e52.21 vace0 has 1953525117 sectors [68228.940404] aoe: 003048dd4d24 e52.20 vace0 has 1953525117 sectors [68228.940408] aoe: 003048dd4d24 e52.19 vace0 has 1953525117 sectors [68228.940413] aoe: 003048dd4d24 e52.18 vace0 has 1953525117 sectors [68228.940418] aoe: 003048dd4d24 e52.17 vace0 has 1953525117 sectors [68228.940423] aoe: 003048dd4d24 e52.16 vace0 has 1953525117 sectors [68228.940427] aoe: 003048dd4d24 e52.15
Re: [Aoetools-discuss] end_request: I/O error
ok...I'm totally lost... :) $ ethtool -k em1 Features for em1: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: on tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: on tx-checksum-fcoe-crc: on [fixed] tx-checksum-sctp: on scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: on rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: on highdma: on [fixed] rx-vlan-filter: on vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: on [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] tx-mpls-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: on loopback: off [fixed] rx-fcs: off [fixed] rx-all: off tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off Daofeng On Tue, Jun 16, 2015 at 12:55 PM, Daofeng Li lid...@gmail.com wrote: Thanks Adi, I'll try. best, Daofeng On Tue, Jun 16, 2015 at 12:01 PM, Adi Kriegisch a...@cg.tuwien.ac.at wrote: Hey! Honestly..I don't know what those dropped packets, it;s just a new installed system. I see... depending on what services are running on that network, this might be some avahi messages or similar stuff; in case this is a storage-only network and there is no other stuff floating around, this may well be the cause for the issues... Do you mean my MTU 9000 is too high? No, I don't think so. Maybe some of the ethernet offloading stuff in the NIC is unable to handle larger packets? You may check with 'ethtool -k em1' what is enabled and may try to selectively disable and retest... -- Adi -- ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] end_request: I/O error
Hey! I was trying to relocating my storage from 1 server to another, the old one has aoe driver version 81 installed, the new has 85 installed, I see this error message on the new system only. Is there a way to fix them? [...] Jun 15 17:11:01 cluster kernel: [ 1243.965424] end_request: I/O error, dev etherd/e50.23, sector 1953523728 [...] # aoe-stat e50.23 1000.204GBem1 8704 up [...] root@cluster:/home/d# ifconfig em1 Link encap:Ethernet HWaddr 5c:f9:dd:b9:48:a8 inet addr:10.200.0.20 Bcast:10.200.0.255 Mask:255.255.255.0 inet6 addr: fe80::5ef9:ddff:feb9:48a8/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:392548 errors:0 dropped:337 overruns:0 frame:0 TX packets:311526 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:164961826 (164.9 MB) TX bytes:20663984 (20.6 MB) Strange. Do you have any idea what the dropped packets (337 above) are? Other than that, are there any offloading features (ethtool) enabled on the nic that cannot reliably deal with jumbo frames? -- Adi -- ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Re: [Aoetools-discuss] end_request: I/O error
Adi, do you think aoetools-36/aoesancheck is a good next step? On 06/16/2015 01:57 PM, Daofeng Li wrote: ok...I'm totally lost... :) $ ethtool -k em1 Features for em1: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: on tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: on tx-checksum-fcoe-crc: on [fixed] tx-checksum-sctp: on scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: on rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: on highdma: on [fixed] rx-vlan-filter: on vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: on [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] tx-mpls-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: on loopback: off [fixed] rx-fcs: off [fixed] rx-all: off tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off Daofeng On Tue, Jun 16, 2015 at 12:55 PM, Daofeng Li lid...@gmail.com mailto:lid...@gmail.com wrote: Thanks Adi, I'll try. best, Daofeng On Tue, Jun 16, 2015 at 12:01 PM, Adi Kriegisch a...@cg.tuwien.ac.at mailto:a...@cg.tuwien.ac.at wrote: Hey! Honestly..I don't know what those dropped packets, it;s just a new installed system. I see... depending on what services are running on that network, this might be some avahi messages or similar stuff; in case this is a storage-only network and there is no other stuff floating around, this may well be the cause for the issues... Do you mean my MTU 9000 is too high? No, I don't think so. Maybe some of the ethernet offloading stuff in the NIC is unable to handle larger packets? You may check with 'ethtool -k em1' what is enabled and may try to selectively disable and retest... -- Adi -- ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss -- ___ Aoetools-discuss mailing list Aoetools-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
