Bug#764162: linux-image-3.16-2-kirkwood: [regression 3.14->3.16] file data corruption, via network
On Tue, 2014-11-04 at 08:51 +, Ian Campbell wrote: > On Fri, 2014-10-31 at 14:29 +0100, Julien D'Ascenzio wrote: > > Le vendredi 31 octobre 2014 à 09:50 +, Ian Campbell a écrit : > > > On Sun, 2014-10-19 at 14:27 +0100, Ian Campbell wrote: > > > > > You could inactivate this feature manually with: > > > > > ethtool -K eth0 tso off > > > > > > > > > > I'm in contact with the developer of this feature. Tell me if this > > > > > command resolve your problem > > > > > > > > Excellent, please let us know how you get on (feel free to CC the bug > > > > too if you want). > > > > > > Did you make any progress with the developer? > > > > > > In the changes to this driver since v3.16 I see > > > commit 817dbfa5d1bc276a72c1a577310382008e8aca0a > > > Author: Vlad Yasevich > > > Date: Mon Aug 25 10:34:54 2014 -0400 > > > > > > mvneta: Fix TSO and checksum for non-acceleration vlan traffic > > > > > > which might be potentially interesting, except no one on this bug has > > > mentioned VLANs at any point. > > > > > > I don't see anything else of interest. > > > > > > Ian. > > > > > > > I don't have more information. They seems not to understand why there > > are data corrupt with this soc. The only solution at this time is to > > inactivate tso or apply this patch, which makes TSO disabled by default: > > I've now managed to update my qnap box too and can trivially see file > corruption over NFS. Unless anyone has any ideas or suggested fixes I'm > going to apply this patch to disable TSO by default to the Debian > kernel. I've just seen https://patchwork.ozlabs.org/patch/405792/ so I'm going to go ahead for now. Ian. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#764162: linux-image-3.16-2-kirkwood: [regression 3.14->3.16] file data corruption, via network
On Fri, 2014-10-31 at 14:29 +0100, Julien D'Ascenzio wrote: > Le vendredi 31 octobre 2014 à 09:50 +, Ian Campbell a écrit : > > On Sun, 2014-10-19 at 14:27 +0100, Ian Campbell wrote: > > > > You could inactivate this feature manually with: > > > > ethtool -K eth0 tso off > > > > > > > > I'm in contact with the developer of this feature. Tell me if this > > > > command resolve your problem > > > > > > Excellent, please let us know how you get on (feel free to CC the bug > > > too if you want). > > > > Did you make any progress with the developer? > > > > In the changes to this driver since v3.16 I see > > commit 817dbfa5d1bc276a72c1a577310382008e8aca0a > > Author: Vlad Yasevich > > Date: Mon Aug 25 10:34:54 2014 -0400 > > > > mvneta: Fix TSO and checksum for non-acceleration vlan traffic > > > > which might be potentially interesting, except no one on this bug has > > mentioned VLANs at any point. > > > > I don't see anything else of interest. > > > > Ian. > > > > I don't have more information. They seems not to understand why there > are data corrupt with this soc. The only solution at this time is to > inactivate tso or apply this patch, which makes TSO disabled by default: I've now managed to update my qnap box too and can trivially see file corruption over NFS. Unless anyone has any ideas or suggested fixes I'm going to apply this patch to disable TSO by default to the Debian kernel. > > diff --git a/drivers/net/ethernet/marvell/mv643xx_eth.c > b/drivers/net/ethernet/marvell/mv643xx_eth.c > index 693b5b5..8e6df75 100644 > --- a/drivers/net/ethernet/marvell/mv643xx_eth.c > +++ b/drivers/net/ethernet/marvell/mv643xx_eth.c > @@ -3113,11 +3113,11 @@ static int mv643xx_eth_probe(struct > platform_device *pdev) > dev->watchdog_timeo = 2 * HZ; > dev->base_addr = 0; > > - dev->features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_TSO; > + dev->features = NETIF_F_SG | NETIF_F_IP_CSUM; > dev->vlan_features = dev->features; > > dev->features |= NETIF_F_RXCSUM; > - dev->hw_features = dev->features; > + dev->hw_features = dev->features | NETIF_F_TSO; > > dev->priv_flags |= IFF_UNICAST_FLT; > dev->gso_max_segs = MV643XX_MAX_TSO_SEGS; > > I CC my contact > > Julien D'Ascenzio > > -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#764162: linux-image-3.16-2-kirkwood: [regression 3.14->3.16] file data corruption, via network
Le vendredi 31 octobre 2014 à 09:50 +, Ian Campbell a écrit : > On Sun, 2014-10-19 at 14:27 +0100, Ian Campbell wrote: > > > You could inactivate this feature manually with: > > > ethtool -K eth0 tso off > > > > > > I'm in contact with the developer of this feature. Tell me if this > > > command resolve your problem > > > > Excellent, please let us know how you get on (feel free to CC the bug > > too if you want). > > Did you make any progress with the developer? > > In the changes to this driver since v3.16 I see > commit 817dbfa5d1bc276a72c1a577310382008e8aca0a > Author: Vlad Yasevich > Date: Mon Aug 25 10:34:54 2014 -0400 > > mvneta: Fix TSO and checksum for non-acceleration vlan traffic > > which might be potentially interesting, except no one on this bug has > mentioned VLANs at any point. > > I don't see anything else of interest. > > Ian. > An other thread about this bug: http://archlinuxarm.org/forum/viewtopic.php?f=9&t=7692 -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#764162: linux-image-3.16-2-kirkwood: [regression 3.14->3.16] file data corruption, via network
Le vendredi 31 octobre 2014 à 09:50 +, Ian Campbell a écrit : > On Sun, 2014-10-19 at 14:27 +0100, Ian Campbell wrote: > > > You could inactivate this feature manually with: > > > ethtool -K eth0 tso off > > > > > > I'm in contact with the developer of this feature. Tell me if this > > > command resolve your problem > > > > Excellent, please let us know how you get on (feel free to CC the bug > > too if you want). > > Did you make any progress with the developer? > > In the changes to this driver since v3.16 I see > commit 817dbfa5d1bc276a72c1a577310382008e8aca0a > Author: Vlad Yasevich > Date: Mon Aug 25 10:34:54 2014 -0400 > > mvneta: Fix TSO and checksum for non-acceleration vlan traffic > > which might be potentially interesting, except no one on this bug has > mentioned VLANs at any point. > > I don't see anything else of interest. > > Ian. > I don't have more information. They seems not to understand why there are data corrupt with this soc. The only solution at this time is to inactivate tso or apply this patch, which makes TSO disabled by default: diff --git a/drivers/net/ethernet/marvell/mv643xx_eth.c b/drivers/net/ethernet/marvell/mv643xx_eth.c index 693b5b5..8e6df75 100644 --- a/drivers/net/ethernet/marvell/mv643xx_eth.c +++ b/drivers/net/ethernet/marvell/mv643xx_eth.c @@ -3113,11 +3113,11 @@ static int mv643xx_eth_probe(struct platform_device *pdev) dev->watchdog_timeo = 2 * HZ; dev->base_addr = 0; - dev->features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_TSO; + dev->features = NETIF_F_SG | NETIF_F_IP_CSUM; dev->vlan_features = dev->features; dev->features |= NETIF_F_RXCSUM; - dev->hw_features = dev->features; + dev->hw_features = dev->features | NETIF_F_TSO; dev->priv_flags |= IFF_UNICAST_FLT; dev->gso_max_segs = MV643XX_MAX_TSO_SEGS; I CC my contact Julien D'Ascenzio -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#764162: linux-image-3.16-2-kirkwood: [regression 3.14->3.16] file data corruption, via network
On Sun, 2014-10-19 at 14:27 +0100, Ian Campbell wrote: > > You could inactivate this feature manually with: > > ethtool -K eth0 tso off > > > > I'm in contact with the developer of this feature. Tell me if this > > command resolve your problem > > Excellent, please let us know how you get on (feel free to CC the bug > too if you want). Did you make any progress with the developer? In the changes to this driver since v3.16 I see commit 817dbfa5d1bc276a72c1a577310382008e8aca0a Author: Vlad Yasevich Date: Mon Aug 25 10:34:54 2014 -0400 mvneta: Fix TSO and checksum for non-acceleration vlan traffic which might be potentially interesting, except no one on this bug has mentioned VLANs at any point. I don't see anything else of interest. Ian. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#764162: linux-image-3.16-2-kirkwood: [regression 3.14->3.16] file data corruption, via network
On Sun, 19 Oct 2014 14:44:11 +0200 Julien D'Ascenzio wrote: > Hi, > > Since v3.16 kernel a tso feature was introduced in the driver of marvell > ethernet. This feature seems to work badly. > > You could inactivate this feature manually with: > ethtool -K eth0 tso off > > I'm in contact with the developer of this feature. Tell me if this > command resolve your problem > > Bye > > I was suffering from this corruption of data sent over network on my Qnap TS-219P II when running linux-image-3.16-3-kirkwood. The command (ethtool -K eth0 tso off) has corrected the problem. Thanks!
Bug#764162: linux-image-3.16-2-kirkwood: [regression 3.14->3.16] file data corruption, via network
On Sun, 2014-10-19 at 14:44 +0200, Julien D'Ascenzio wrote: > Since v3.16 kernel a tso feature was introduced in the driver of marvell > ethernet. This feature seems to work badly. 3.16-1~exp1 was reported as not exhibiting the issue, but maybe that was just a quick test which got lucky. So this is nonetheless a good tip and something worth trying, thanks! > You could inactivate this feature manually with: > ethtool -K eth0 tso off > > I'm in contact with the developer of this feature. Tell me if this > command resolve your problem Excellent, please let us know how you get on (feel free to CC the bug too if you want). Ian. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#764162: linux-image-3.16-2-kirkwood: [regression 3.14->3.16] file data corruption, via network
Hi, Since v3.16 kernel a tso feature was introduced in the driver of marvell ethernet. This feature seems to work badly. You could inactivate this feature manually with: ethtool -K eth0 tso off I'm in contact with the developer of this feature. Tell me if this command resolve your problem Bye -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#764162: linux-image-3.16-2-kirkwood: [regression 3.14->3.16] file data corruption, via network
On Sun, 2014-10-12 at 23:46 +0200, Svenska wrote: > > If you were able to try 3.16-1~exp1[0] > > from snapshot.debian.org that might help rule that out. > > I downloaded linux-image-3.16-trunk-kirkwood_3.16-1~exp1_armel.deb from > your link and flashed it into the NAS. I have not done a deeper > analysis, but it looks like the problem is gone. > > For now, I will revert back to the known-working 3.14-2 (3.14.15-2) > anyway, but if you want me to test another binary image, I could do that > if given a week or two. > > Hope this helps. Thanks, it narrows it down to a change on the Debian side rather than a generic 3.16 upgrade breakage. Looking at the changelog the most plausible looking candidate is still the enabling of CONFIG_HIGHMEM. Your system has far less RAM than I would expect to need highmem though. If you are comfortable building a kernel for yourself (see [0] for details) then it might be interesting to try rebuilding without CONFIG_HIGHMEM. Ian. [0] http://kernel-handbook.alioth.debian.org/ch-common-tasks.html#s-common-official -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#764162: linux-image-3.16-2-kirkwood: [regression 3.14->3.16] file data corruption, via network
If you were able to try 3.16-1~exp1[0] from snapshot.debian.org that might help rule that out. I downloaded linux-image-3.16-trunk-kirkwood_3.16-1~exp1_armel.deb from your link and flashed it into the NAS. I have not done a deeper analysis, but it looks like the problem is gone. For now, I will revert back to the known-working 3.14-2 (3.14.15-2) anyway, but if you want me to test another binary image, I could do that if given a week or two. Hope this helps. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#764162: linux-image-3.16-2-kirkwood: [regression 3.14->3.16] file data corruption, via network
On Mon, 2014-10-06 at 01:21 +0200, Svenska wrote:
> Package: src:linux
> Version: 3.16.3-2
> Severity: serious
> Justification: may cause silent data corruption
>
> Hello,
>
> after upgrading the kernel of my NAS to 3.16-2-kirkwood, I noticed
> corrupt data on my files. The NAS works as a DHCP client on its single
> LAN port, the two hard drives run in a RAID1 configuration. Both
> drives are fine according to mdadm and smartctl. Kernel logs in both
> cases have not been eye-catching. Samba is configured with "unix
> charset = ISO-8859-1".
>
> When copying data from the NAS to some other device using CIFS or
> Samba (the other devices run jessie/amd64 or wheezy/i686), the files
> are partly corrupt. The reported md5 checksums of files change every
> time the files are read. (And they differ from the checksums as stored
> on disk as well.) The files are not completely destroyed: ZIP archives
> show CRC errors only for some files in them, and video files remain
> generally playable, with lots of audio/video errors and crashing
> players.
>
> Shell access using SSH generally works. Trying to copy files with scp
> or sftp breaks the connection ("invalid MAC packets" or similar). File
> listing and mounting with sshfs work. Trying to copy files via sshfs
> results in a "Socket not connected" error, and the mount point is
> gone.
>
> As far as I can see, the on-disk data seems to be fine, as do files
> copied to the NAS while running the corrupt kernel. I am unable to
> check all new files, though - there might have been silent data
> corruption on new files.
It's sounding like the issue might be corruption on the networking path
rather than the disk path then?
> Data corruption on NAS devices is a serious issue for me.
> Reverting back to 3.14-2-kirkwood fixed the problems.
Which exact version did you revert to? Looking at debian/changelog since
the first 3.14 version I don't see much specific to kirkwood. At some
point there were some changes with the network PHY but they predate 3.14
IIRC.
The only thing which jumps out is "[armel/kirkwood] mm: Enable HIGHMEM
(Closes: #760786)" in 3.16.2-1. If you were able to try 3.16-1~exp1[0]
from snapshot.debian.org that might help rule that out.
If 3.16-1~exp1 still has the issue then I think we are looking at an
upstream regression. It might be worth bisecting a bit over the old
kernels from experimental which can be found on snapshot.d.o e.g. to
narrow down if the issue appeared in 3.15 or 3.16.
After that please could you to take this up with the upstream
developers[1], who are normally very responsive to kirkwood issues.
Thanks,
Ian.
[0] Should be at http://snapshot.debian.org/package/linux/3.16-1~exp1/
[1]
ARM/Marvell Dove/Kirkwood/MV78xx0/Orion SOC support
M: Jason Cooper
M: Andrew Lunn
M: Sebastian Hesselbarth
L: linux-arm-ker...@lists.infradead.org (moderated for non-subscribers)
S: Maintained
>
> Best Regards
>
> -- Package-specific info:
> ** Kernel log: boot messages should be attached
>
> ** Model information
> Hardware : QNAP TS-119/TS-219
> Revision :
>
> ** PCI devices:
> 00:00.0 Host bridge [0600]: Marvell Technology Group Ltd. Device
> [11ab:6282] (rev 01)
> Subsystem: Marvell Technology Group Ltd. Device [11ab:11ab]
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-Latency: 0, Cache Line Size: 32 bytes
> Interrupt: pin A routed to IRQ 9
> Region 0: Memory at (64-bit, prefetchable)
> Capabilities:
>
> 00:01.0 SATA controller [0106]: JMicron Technology Corp. JMB363 SATA/IDE
> Controller [197b:2363] (rev 03) (prog-if 01 [AHCI 1.0])
> Subsystem: JMicron Technology Corp. JMB363 SATA/IDE Controller
> [197b:2363]
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-Latency: 0, Cache Line Size: 32 bytes
> Interrupt: pin A routed to IRQ 9
> Region 5: Memory at e001 (32-bit, non-prefetchable) [size=8K]
> [virtual] Expansion ROM at e000 [disabled] [size=64K]
> Capabilities:
> Kernel driver in use: ahci
>
> 00:01.1 IDE interface [0101]: JMicron Technology Corp. JMB363 SATA/IDE
> Controller [197b:2363] (rev 03) (prog-if 85 [Master SecO PriO])
> Subsystem: JMicron Technology Corp. JMB363 SATA/IDE Controller
> [197b:2363]
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+
> Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-Interrupt: pin B routed to IRQ 9
> Region 0: I/O ports at 1010 [disabled] [size=8]
> Region 1: I/O ports at 1020 [disabled] [size=4]
> Region 2: I/O ports at 1018 [disabled] [size=8]
> Region 3: I/O p
Bug#764162: linux-image-3.16-2-kirkwood: [regression 3.14->3.16] file data corruption, via network
Package: src:linux
Version: 3.16.3-2
Severity: serious
Justification: may cause silent data corruption
Hello,
after upgrading the kernel of my NAS to 3.16-2-kirkwood, I noticed
corrupt data on my files. The NAS works as a DHCP client on its single
LAN port, the two hard drives run in a RAID1 configuration. Both
drives are fine according to mdadm and smartctl. Kernel logs in both
cases have not been eye-catching. Samba is configured with "unix
charset = ISO-8859-1".
When copying data from the NAS to some other device using CIFS or
Samba (the other devices run jessie/amd64 or wheezy/i686), the files
are partly corrupt. The reported md5 checksums of files change every
time the files are read. (And they differ from the checksums as stored
on disk as well.) The files are not completely destroyed: ZIP archives
show CRC errors only for some files in them, and video files remain
generally playable, with lots of audio/video errors and crashing
players.
Shell access using SSH generally works. Trying to copy files with scp
or sftp breaks the connection ("invalid MAC packets" or similar). File
listing and mounting with sshfs work. Trying to copy files via sshfs
results in a "Socket not connected" error, and the mount point is
gone.
As far as I can see, the on-disk data seems to be fine, as do files
copied to the NAS while running the corrupt kernel. I am unable to
check all new files, though - there might have been silent data
corruption on new files.
Data corruption on NAS devices is a serious issue for me.
Reverting back to 3.14-2-kirkwood fixed the problems.
Best Regards
-- Package-specific info:
** Kernel log: boot messages should be attached
** Model information
Hardware: QNAP TS-119/TS-219
Revision:
** PCI devices:
00:00.0 Host bridge [0600]: Marvell Technology Group Ltd. Device
[11ab:6282] (rev 01)
Subsystem: Marvell Technology Group Ltd. Device [11ab:11ab]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 9
Region 0: Memory at (64-bit, prefetchable)
Capabilities:
00:01.0 SATA controller [0106]: JMicron Technology Corp. JMB363 SATA/IDE
Controller [197b:2363] (rev 03) (prog-if 01 [AHCI 1.0])
Subsystem: JMicron Technology Corp. JMB363 SATA/IDE Controller
[197b:2363]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 9
Region 5: Memory at e001 (32-bit, non-prefetchable) [size=8K]
[virtual] Expansion ROM at e000 [disabled] [size=64K]
Capabilities:
Kernel driver in use: ahci
00:01.1 IDE interface [0101]: JMicron Technology Corp. JMB363 SATA/IDE
Controller [197b:2363] (rev 03) (prog-if 85 [Master SecO PriO])
Subsystem: JMicron Technology Corp. JMB363 SATA/IDE Controller
[197b:2363]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
Interrupt: pin B routed to IRQ 9
Region 0: I/O ports at 1010 [disabled] [size=8]
Region 1: I/O ports at 1020 [disabled] [size=4]
Region 2: I/O ports at 1018 [disabled] [size=8]
Region 3: I/O ports at 1024 [disabled] [size=4]
Region 4: I/O ports at 1000 [disabled] [size=16]
Capabilities:
01:00.0 Host bridge [0600]: Marvell Technology Group Ltd. Device
[11ab:6282] (rev 01)
Subsystem: Marvell Technology Group Ltd. Device [11ab:11ab]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 10
Region 0: Memory at (64-bit, prefetchable)
Capabilities:
01:01.0 USB controller [0c03]: Etron Technology, Inc. EJ168 USB 3.0 Host
Controller [1b6f:7023] (rev 01) (prog-if 30 [XHCI])
Subsystem: Etron Technology, Inc. EJ168 USB 3.0 Host Controller
[1b6f:7023]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 10
Region 0: Memory at e800 (64-bit, non-prefetchable) [size=32K]
Capabilities:
Kernel driver in use: xhci_hcd
** USB devices:
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
