Re: NFS write corruption on 8.0-RELEASE
On Fri, Feb 12, 2010 at 04:08:55PM -0800, alan bryan wrote: --- On Fri, 2/12/10, Rick Macklem rmack...@uoguelph.ca wrote: From: Rick Macklem rmack...@uoguelph.ca Subject: Re: NFS write corruption on 8.0-RELEASE To: Dmitry Marakasov amd...@amdmi3.ru Cc: freebsd-hackers@freebsd.org, freebsd-sta...@freebsd.org, John Baldwin j...@freebsd.org Date: Friday, February 12, 2010, 11:12 AM On Fri, 12 Feb 2010, Dmitry Marakasov wrote: I'm planning a massive testing for this weekend, including removing soft mount option and trying linux client/server. Btw, I forgot to mention that I'm experiencing other NFS problems from time to time, including death of a mount (that is, all processes that try to access it freeze; this cures itself in some time with a message server is alive again). Also I've seen another strange thing - not only the mount dies but the network is flooded with NFS traffic. Last time I've seen it quite a while ago, so I don't remember the circumstances and direction of the traffic. There are some patches at: http://people.freebsd.org/~rmacklem that may be relevant if you are using vanilla FreeBSD-8.0. (They're all now in stable/8, but are post-release of 8.0.) rick ___ freebsd-sta...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org This is interesting: I've seen another strange thing - not only the mount dies but the network is flooded with NFS traffic. You might be able to see NFS flooding with: Write file on client. rm the file during the write on the server. The client now gets IO-errors, but keeps trying forever. Maybe it depends on the mechanism the client application uses to write the file. I never reported because my client is an old 7.0-stable system. I originally noticed it when downloading files with seamonkey on my client and mv it on the server to another partition before it was completely downloaded. -- B.Walter be...@bwct.de http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS write corruption on 8.0-RELEASE
* Rick Macklem (rmack...@uoguelph.ca) wrote: Is it the hostname of the server or the client? My guess is that hades.panopticon (or something like that:-) is the Yes, that is the client. As John said, it would be nice to try and narrow it down to client or server side, too. I'm planning a massive testing for this weekend, including removing soft mount option and trying linux client/server. Btw, I forgot to mention that I'm experiencing other NFS problems from time to time, including death of a mount (that is, all processes that try to access it freeze; this cures itself in some time with a message server is alive again). Also I've seen another strange thing - not only the mount dies but the network is flooded with NFS traffic. Last time I've seen it quite a while ago, so I don't remember the circumstances and direction of the traffic. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS write corruption on 8.0-RELEASE
* Oliver Fromme (o...@lurza.secnetix.de) wrote: This is an excerpt from Solaris' mount_nfs(1M) manpage: File systems that are mounted read-write or that con- tain executable files should always be mounted with the hard option. Applications using soft mounted file systems may incur unexpected I/O errors, file corrup- tion, and unexpected program core dumps. The soft option is not recommended. FreeBSD's manual page doesn't contain such a warning, but maybe it should. (It contains a warning not to use soft with NFSv4, though, for different reasons.) Interesting, I'll try disabling it. However now I really wonder why is such dangerous option available (given it's the cause) at all, especially without a notice. Silent data corruption is possibly the worst thing to happen ever. However, without soft option NFS would be a strange thing to use - network problems is kinda inevitable thing, and having all processes locked in a unkillable state (with hard mounts) when it dies is not fun. Or am I wrong? Also note that the nolockd option means that processes on different clients won't see each other's locks. That means that you will get corruption if they rely on locking. I know - I have no processes that use locks on that filesystems. Also there's only a single client. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS write corruption on 8.0-RELEASE
On Thu, 11 Feb 2010, John Baldwin wrote: [good stuff snipped] Case1: single currupted block 3779CF88-3779 (12408 bytes). Data in block is shifted 68 bytes up, loosing first 68 bytes are filling last 68 bytes with garbage. Interestingly, among that garbage is my hostname. Is it the hostname of the server or the client? My guess is that hades.panopticon (or something like that:-) is the client. The garbage is 4 bytes (80 00 80 84) followed by the first part of the RPC header. (Bytes 5-8 vary because they are the xid and then the host name is part of the AUTH_SYS authenticator.) For Case2 and Case3, you see less of it, but it's the same stuff. Why? I have no idea, although it smells like some sort of corruption of the mbuf list. (It would be nice if you could switch to a different net interface/driver. Just a thought, since others don't seem to be seeing this?) As John said, it would be nice to try and narrow it down to client or server side, too. Don't know if this helps or is just noise, rick ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS write corruption on 8.0-RELEASE
On Fri, 12 Feb 2010, Dmitry Marakasov wrote: * Oliver Fromme (o...@lurza.secnetix.de) wrote: This is an excerpt from Solaris' mount_nfs(1M) manpage: File systems that are mounted read-write or that con- tain executable files should always be mounted with the hard option. Applications using soft mounted file systems may incur unexpected I/O errors, file corrup- tion, and unexpected program core dumps. The soft option is not recommended. FreeBSD's manual page doesn't contain such a warning, but maybe it should. (It contains a warning not to use soft with NFSv4, though, for different reasons.) Interesting, I'll try disabling it. However now I really wonder why is such dangerous option available (given it's the cause) at all, especially without a notice. Silent data corruption is possibly the worst thing to happen ever. Tell me about it. :) But in this case I'm not sure I understand. As I understand it, the difference between soft and hard is that in the case of soft, a timeout will result in the operation failing and returning EIO or the like (hence unexpected I/O errors). And if the operation is being done to fault in a mapped page, you'd have to notify the process asynchronously by sending a signal like SIGBUS which it may not be expecting (hence unexpected core dumps). But in what scenario would you see file corruption? Unless you have a buggy program that doesn't check return values from system calls or handles signals in a stupid way, I don't see how this can happen, and I'm not sure what the Sun man page is referring to. -- Nate Eldredge n...@thatsmathematics.com ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS write corruption on 8.0-RELEASE
Dmitry Marakasov wrote: * Oliver Fromme (o...@lurza.secnetix.de) wrote: This is an excerpt from Solaris' mount_nfs(1M) manpage: File systems that are mounted read-write or that con- tain executable files should always be mounted with the hard option. Applications using soft mounted file systems may incur unexpected I/O errors, file corrup- tion, and unexpected program core dumps. The soft option is not recommended. FreeBSD's manual page doesn't contain such a warning, but maybe it should. (It contains a warning not to use soft with NFSv4, though, for different reasons.) Interesting, I'll try disabling it. However now I really wonder why is such dangerous option available (given it's the cause) at all, especially without a notice. Silent data corruption is possibly the worst thing to happen ever. I'm sorry for the confusion ... I do not think that it's the cause for your data corruption, in this particular case. I just mentioned the potential problems with soft mounts because it could cause additional problems for you. (And it's important to know anyhow.) However, without soft option NFS would be a strange thing to use - network problems is kinda inevitable thing, and having all processes locked in a unkillable state (with hard mounts) when it dies is not fun. Or am I wrong? Well, this is what happens if the network hangs: 1. With hard mounts (the default), processes that access NFS shares are locked for as long as the network is down. 2. With soft mounts, binaries can coredump, and many programs won't notice that write access just failed which leads to file corruption. Personally I definitely prefer the first. Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd I have stopped reading Stephen King novels. Now I just read C code instead. -- Richard A. O'Keefe ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS write corruption on 8.0-RELEASE
* Oliver Fromme (o...@lurza.secnetix.de) wrote: I'm sorry for the confusion ... I do not think that it's the cause for your data corruption, in this particular case. I just mentioned the potential problems with soft mounts because it could cause additional problems for you. (And it's important to know anyhow.) Oh, then I really misunderstood. If the curruption implied is like when you copy a file via NFS and the net goes down, and in case of soft mount you have half of a file (read: corruption), while with hard mount the copy process will finish when the net is back up, that's definitely OK and expected. Well, this is what happens if the network hangs: 1. With hard mounts (the default), processes that access NFS shares are locked for as long as the network is down. 2. With soft mounts, binaries can coredump, and many programs won't notice that write access just failed which leads to file corruption. Personally I definitely prefer the first. Yeah, but I have mostly desktop-(NAS w/torrents) setup so I prefer the second. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS write corruption on 8.0-RELEASE
On Fri, 12 Feb 2010, Dmitry Marakasov wrote: Interesting, I'll try disabling it. However now I really wonder why is such dangerous option available (given it's the cause) at all, especially without a notice. Silent data corruption is possibly the worst thing to happen ever. I doubt that the data corruption you are seeing would be because of soft. soft will cause various problems w.r.t. consistency, but in the case of a write through the buffer cache, I think it will leave the buffer dirty and eventually it will get another write attempt. However, without soft option NFS would be a strange thing to use - network problems is kinda inevitable thing, and having all processes locked in a unkillable state (with hard mounts) when it dies is not fun. Or am I wrong? Well, using NFS over an unreliable network is going to cause grief sooner or later. The problem is that POSIX apps. don't expect I/O system calls to fail with EIO and generally don't handle that gracefully. For the future, I think umount -F (a forced dismount that accepts data loss) is the best compromise, since at least then a sysadmin knows that data corruption could have occurred when they do it and can choose to wait until the network is fixed as an alternative to the corruption? rick ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS write corruption on 8.0-RELEASE
Dmitry Marakasov wrote: Oh, then I really misunderstood. If the curruption implied is like when you copy a file via NFS and the net goes down, and in case of soft mount you have half of a file (read: corruption), while with hard mount the copy process will finish when the net is back up, that's definitely OK and expected. Of course it depends what kinds of programs you run. If you run a database, it can become corrupted to the point that you can't start it anymore, so you have to restore it from the latest backup. Been there, done that. Another example, this happened to a friend of mine: After a network outage his Opera browser didn't work anymore. He had to remove his ~/.opera directory to get it working again (and he lost all his settings). His home directory was soft-mounted, but he removed the soft option after that incident. Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd The last good thing written in C was Franz Schubert's Symphony number 9. -- Erwin Dieterich ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS write corruption on 8.0-RELEASE
On Fri, 12 Feb 2010, Dmitry Marakasov wrote: * Oliver Fromme (o...@lurza.secnetix.de) wrote: I'm sorry for the confusion ... I do not think that it's the cause for your data corruption, in this particular case. I just mentioned the potential problems with soft mounts because it could cause additional problems for you. (And it's important to know anyhow.) Oh, then I really misunderstood. If the curruption implied is like when you copy a file via NFS and the net goes down, and in case of soft mount you have half of a file (read: corruption), while with hard mount the copy process will finish when the net is back up, that's definitely OK and expected. The problem is that it can't distinguish between slow network/server and partitioned/failed network. In your case (one client) it may work out ok. (I can't remember how long it takes to timeout and give up for soft.) For many clients talking to an NFS server, the NFS server's response time can degrade to the point where soft mounted clients start timing out and that can get ugly. rick ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS write corruption on 8.0-RELEASE
--- On Fri, 2/12/10, Rick Macklem rmack...@uoguelph.ca wrote: From: Rick Macklem rmack...@uoguelph.ca Subject: Re: NFS write corruption on 8.0-RELEASE To: Dmitry Marakasov amd...@amdmi3.ru Cc: freebsd-hackers@freebsd.org, freebsd-sta...@freebsd.org, John Baldwin j...@freebsd.org Date: Friday, February 12, 2010, 11:12 AM On Fri, 12 Feb 2010, Dmitry Marakasov wrote: I'm planning a massive testing for this weekend, including removing soft mount option and trying linux client/server. Btw, I forgot to mention that I'm experiencing other NFS problems from time to time, including death of a mount (that is, all processes that try to access it freeze; this cures itself in some time with a message server is alive again). Also I've seen another strange thing - not only the mount dies but the network is flooded with NFS traffic. Last time I've seen it quite a while ago, so I don't remember the circumstances and direction of the traffic. There are some patches at: http://people.freebsd.org/~rmacklem that may be relevant if you are using vanilla FreeBSD-8.0. (They're all now in stable/8, but are post-release of 8.0.) rick ___ freebsd-sta...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org This is interesting: I've seen another strange thing - not only the mount dies but the network is flooded with NFS traffic. Rick - this sounds very similar to the issues I was seeing (and reported in the thread on freebsd-stable Zombie NFS writing from FreeBSD clients to FreeBSD 8.0 server with ZFS. For the record - I updated to the latest 8-Stable and that still didn't cure my issues. I was originally on hard mounts on udp, tried soft and TCP too, nothing solved it. So, a few days ago I switched to using samba and mount_smbfs instead and am now running 3 days without a crash or any network traffic/load issues. (same machine, same ZFS disks, etc...) Luckily it wasn't too painful to make the change. When I have more time I'd like to retry NFS. --Alan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS write corruption on 8.0-RELEASE
On Wednesday 10 February 2010 12:43:38 pm Dmitry Marakasov wrote: Hi! I think I've reported that before, the I thought it's been fixed, however I still get data corruptions when writing on NFS volumes. Now I wonder - is nobody really using NFS, or do I have that much of uncommon setup, or this is some kind of local problem? Client: 8.0-RELEASE i386 Server: 8.0-RELEASE amd64 mount options: nfs rw,nosuid,noexec,nfsv3,intr,soft,tcp,bg,nolockd Server has ZFS, but the same thing happens when sharing UFS placed on md(4). I've prepared a special 1GB file to determine the details of corruption: it's filled with 32-bit numbers each equal to it's own offset in the file. That is, like that: 00 00 00 00 04 00 00 00 08 00 00 00 0c 00 00 00 || 0010 10 00 00 00 14 00 00 00 18 00 00 00 1c 00 00 00 || 0020 20 00 00 00 24 00 00 00 28 00 00 00 2c 00 00 00 | ...$...(...,...| 0030 30 00 00 00 34 00 00 00 38 00 00 00 3c 00 00 00 |0...4...8..| I've copied that file over NFS from client to server around 50 times, and got 3 corruptions on 8th, 28th and 36th copies. Case1: single currupted block 3779CF88-3779 (12408 bytes). Data in block is shifted 68 bytes up, loosing first 68 bytes are filling last 68 bytes with garbage. Interestingly, among that garbage is my hostname. Is it the hostname of the server or the client? Case2: single currupted block 2615CFA0-2615 (12384 bytes). Data is shifted by 44 bytes in the same way. Case3: single currepted block 3AA947A8-3AA97FFF (14424 bytes). Data is shifted by 36 bytes in the same way. Any ideas? PS. Diffs of corrupted blocks in a text format are here: http://people.freebsd.org/~amdmi3/diff.1.txt http://people.freebsd.org/~amdmi3/diff.2.txt http://people.freebsd.org/~amdmi3/diff.3.txt Can you reproduce this using a non-FreeBSD server with a FreeBSD client or a non-FreeBSD client with a FreeBSD server? That would narrow down the breakage to either the client or the server. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
NFS write corruption on 8.0-RELEASE
Hi! I think I've reported that before, the I thought it's been fixed, however I still get data corruptions when writing on NFS volumes. Now I wonder - is nobody really using NFS, or do I have that much of uncommon setup, or this is some kind of local problem? Client: 8.0-RELEASE i386 Server: 8.0-RELEASE amd64 mount options: nfs rw,nosuid,noexec,nfsv3,intr,soft,tcp,bg,nolockd Server has ZFS, but the same thing happens when sharing UFS placed on md(4). I've prepared a special 1GB file to determine the details of corruption: it's filled with 32-bit numbers each equal to it's own offset in the file. That is, like that: 00 00 00 00 04 00 00 00 08 00 00 00 0c 00 00 00 || 0010 10 00 00 00 14 00 00 00 18 00 00 00 1c 00 00 00 || 0020 20 00 00 00 24 00 00 00 28 00 00 00 2c 00 00 00 | ...$...(...,...| 0030 30 00 00 00 34 00 00 00 38 00 00 00 3c 00 00 00 |0...4...8..| I've copied that file over NFS from client to server around 50 times, and got 3 corruptions on 8th, 28th and 36th copies. Case1: single currupted block 3779CF88-3779 (12408 bytes). Data in block is shifted 68 bytes up, loosing first 68 bytes are filling last 68 bytes with garbage. Interestingly, among that garbage is my hostname. Case2: single currupted block 2615CFA0-2615 (12384 bytes). Data is shifted by 44 bytes in the same way. Case3: single currepted block 3AA947A8-3AA97FFF (14424 bytes). Data is shifted by 36 bytes in the same way. Any ideas? PS. Diffs of corrupted blocks in a text format are here: http://people.freebsd.org/~amdmi3/diff.1.txt http://people.freebsd.org/~amdmi3/diff.2.txt http://people.freebsd.org/~amdmi3/diff.3.txt -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amd...@amdmi3.ru ..: jabber: amd...@jabber.ruhttp://www.amdmi3.ru ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS write corruption on 8.0-RELEASE
Dmitry Marakasov amd...@amdmi3.ru wrote: I think I've reported that before, the I thought it's been fixed, however I still get data corruptions when writing on NFS volumes. Now I wonder - is nobody really using NFS, or do I have that much of uncommon setup, or this is some kind of local problem? NFS works fine for me. I'm using -stable, not -release, though. Client: 8.0-RELEASE i386 Server: 8.0-RELEASE amd64 mount options: nfs rw,nosuid,noexec,nfsv3,intr,soft,tcp,bg,nolockd I recommend not using the soft option. This is an excerpt from Solaris' mount_nfs(1M) manpage: File systems that are mounted read-write or that con- tain executable files should always be mounted with the hard option. Applications using soft mounted file systems may incur unexpected I/O errors, file corrup- tion, and unexpected program core dumps. The soft option is not recommended. FreeBSD's manual page doesn't contain such a warning, but maybe it should. (It contains a warning not to use soft with NFSv4, though, for different reasons.) Also note that the nolockd option means that processes on different clients won't see each other's locks. That means that you will get corruption if they rely on locking. Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd Perl will consistently give you what you want, unless what you want is consistency. -- Larry Wall ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
