Re: Data corruption over NFS in -current
Martin Cracauer wrote:
> More findings.
>
> Reminder, with the original report I found:
> - files for no reason changing ownership and group to
> root/
> - data corruption as in inserting binary junk obviously from ports
> - data corruption as in malformed ascii text that might be a bug I
> have in my code that is only exposed in FreeBSD
>
> I ran the script on a Linux machine in the same situation again the
> same
> NFS server, it worked fine. I haven't look at blocksizes, NFS
> versions etc in play yet.
>
> I ran with oldnfs (reboot), which showed only the third problem.
>
> I re-ran with newfs (reboot) which worked (all three problems absent).
>
> I then started building ports/land/gcc47 at the same time as I
> re-started my crazy script and it too only a few seconds for an
> unexpected ownership to root to occur.
>
> My next steps are:
> - trying block sizes and other parameters, maybe use a different NFS
> version with the Linux client. My NFS server is newly upgraded to
> Linux kernel 3.1.5
> - running my script on a FreeBSD host with local disk to see whether
> problem #3 is a general problem that appears or is exposed only on
> FreeBSD
> - capture tcpdump as mentioned earlier
>
> I will probably have to turn debug off since this script run is
> dominated by system time now and gets 10x slower as it is now.
>
While poking around (partly related to this and partly related to
the NFSv4.1 pNFS client work), I came across an ugly bug in the
way the new NFS client handled "system operations". ("system operations"
are mainly NFSv4 Ops that manage state, such as Renew, which renews
a lease for the open/lock state. Another case of this was the NFSv3 statfs
when it did a Getattr because the server did not provide post operation
attributes in the reply.) It turns out that at least some Linux NFSv3
servers are in this category and the fact that Martin was doing a large
number of StatFS RPCs was indeed relevent.
Anyhow, the patch to fix the above seems to have resolved Martin's
problem. The patch is needed for the new NFS client if you are using
NFSv4 mounts or NFSv3 mounts against non-FreeBSD servers that don't
provide post-op attributes in the Statfs RPC reply. (FreeBSD servers
do provide post-op attributes, at least some Linux servers do not and
I don't know about others. You could check by capturing the packets
for a "df" and then looking at Statfs RPC reply in wireshark.) Without
the patch, there will be intermittent permission failures, since the
wrong credentials get used for an RPC.
The patch is here and should be in head soon:
http://people.freebsd.org/~rmacklem/authcred.patch
Thanks go to Martin for pursuing this.
rick
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [panic] intr_event_execute_handlers() - Corrupted DWARF expression
On Thursday, January 19, 2012 11:02:57 am Glen Barber wrote: > On Thu, Jan 19, 2012 at 10:50:45AM -0500, John Baldwin wrote: > > On Wednesday, January 18, 2012 5:01:37 pm Glen Barber wrote: > > > Hi, > > > > > > I'm running -CURRENT from about 5 days ago: > > > > > > nucleus# uname -a > > > FreeBSD nucleus 10.0-CURRENT FreeBSD 10.0-CURRENT #3 r230037M: Fri Jan > > > 13 17:48:14 EST 2012 gjb@nucleus:/usr/obj/usr/src/sys/NUCLEUS amd64 > > > > > > (The 'M' is kib's DRM patches for Intel GPU.) > > > > > > So far, I haven't had much problem with this laptop, but just had the > > > machine panic. > > > > > > I have kgdb output attached, and I'll be happy to provide whatever > > > additional information that may be needed. > > > > > > I have core.txt.N available here: > > > > > > http://people.freebsd.org/~gjb/core.txt > > > > In kgdb, can you go to frame 6 and 'p td->td_lock'. If that is non-null, > > can > > you do 'p *td->td_lock'? > > > > Sure, script(1) output is attached. Hmm, I don't think td->td_lock is ever supposed to be NULL. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
posix_fadvise noreuse disables file caching
Hi, I recently noticed that multimedia/vlc generates a lot of disk IO when playing media files. For instance, when playing a 320kbps mp3 gstat reports about 1250kBps (=1kbps). That's quite a lot of overhead. It turns out that vlc sets POSIX_FADV_NOREUSE on the entire file and reads in chunks of 1028 bytes. FreeBSD implements NOREUSE as if O_DIRECT was specified during open(2), i.e. it disables all caching. That means every 1028 byte read turns into a 32KiB read (new default block size in 9.0) which explains the above numbers. I've copied the relevant vlc code below (modules/access/file.c:Open()). It's interesting to see that on OSX it sets F_NOCACHE which disables caching too, but combined with F_RDAHEAD there's still read-ahead caching. I don't think POSIX intended for NOREUSE to mean O_DIRECT. It should still cache data (and even do read-ahead if F_RDAHEAD is specified), and once data is fetched from the cache, it can be marked WONTNEED. Is it possible to implement it this way, or if not to just ignore the NOREUSE hint for now? /* Demuxers will need the beginning of the file for probing. */ posix_fadvise (fd, 0, 4096, POSIX_FADV_WILLNEED); /* In most cases, we only read the file once. */ posix_fadvise (fd, 0, 0, POSIX_FADV_NOREUSE); #if defined(HAVE_FCNTL) /* We'd rather use any available memory for reading ahead * than for caching what we've already seen/heard */ # if defined(F_RDAHEAD) fcntl (fd, F_RDAHEAD, 1); # endif # if defined(F_NOCACHE) fcntl (fd, F_NOCACHE, 1); # endif #endif signature.asc Description: This is a digitally signed message part.
Re: [panic] intr_event_execute_handlers() - Corrupted DWARF expression
On Thu, Jan 19, 2012 at 10:50:45AM -0500, John Baldwin wrote: > On Wednesday, January 18, 2012 5:01:37 pm Glen Barber wrote: > > Hi, > > > > I'm running -CURRENT from about 5 days ago: > > > > nucleus# uname -a > > FreeBSD nucleus 10.0-CURRENT FreeBSD 10.0-CURRENT #3 r230037M: Fri Jan > > 13 17:48:14 EST 2012 gjb@nucleus:/usr/obj/usr/src/sys/NUCLEUS amd64 > > > > (The 'M' is kib's DRM patches for Intel GPU.) > > > > So far, I haven't had much problem with this laptop, but just had the > > machine panic. > > > > I have kgdb output attached, and I'll be happy to provide whatever > > additional information that may be needed. > > > > I have core.txt.N available here: > > > > http://people.freebsd.org/~gjb/core.txt > > In kgdb, can you go to frame 6 and 'p td->td_lock'. If that is non-null, can > you do 'p *td->td_lock'? > Sure, script(1) output is attached. Thanks. Glen Script started on Thu Jan 19 10:56:46 2012 nucleus# kgdb kernel.debug /var/crash/vmcore.4 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: ACPI Warning: Large Reference Count (0x806) in object 0xfe000449fc80 (20120111/utdelete-491) ACPI Warning: Large Reference Count (0x807) in object 0xfe000449fc80 (20120111/utdelete-491) kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 04 fault virtual address = 0x18 fault code = supervisor read data, page not present instruction pointer = 0x20:0x805c2098 stack pointer = 0x28:0xff8000269a50 frame pointer = 0x28:0xff8000269aa0 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= resume, IOPL = 0 current process = 12 (swi4: clock) trap number = 12 panic: page fault cpuid = 2 KDB: stack backtrace: #0 0x806009ce at kdb_backtrace+0x5e #1 0x805d06a8 at panic+0x1d8 #2 0x8081a000 at trap_fatal+0x290 #3 0x8081a63d at trap+0x29d #4 0x808063bf at calltrap+0x8 #5 0x80609bdd at sleepq_timeout+0x1d #6 0x805e238f at softclock+0x29f #7 0x805ab904 at intr_event_execute_handlers+0x64 #8 0x805ac567 at ithread_loop+0x97 #9 0x805a975d at fork_exit+0x11d #10 0x808068ee at fork_trampoline+0xe Uptime: 2h42m5s Dumping 2511 out of 7846 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% kgdb: kvm_read: invalid address (0x64) Reading symbols from /boot/kernel/vesa.ko...Reading symbols from /boot/kernel/vesa.ko.symbols...done. done. Loaded symbols for /boot/kernel/vesa.ko Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done. done. Loaded symbols for /boot/kernel/linux.ko Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from /boot/kernel/coretemp.ko.symbols...done. done. Loaded symbols for /boot/kernel/coretemp.ko Reading symbols from /boot/kernel/sem.ko...Reading symbols from /boot/kernel/sem.ko.symbols...done. done. Loaded symbols for /boot/kernel/sem.ko Reading symbols from /boot/kernel/i915.ko...Reading symbols from /boot/kernel/i915.ko.symbols...done. done. Loaded symbols for /boot/kernel/i915.ko Reading symbols from /boot/kernel/iicbb.ko...Reading symbols from /boot/kernel/iicbb.ko.symbols...done. done. Loaded symbols for /boot/kernel/iicbb.ko Reading symbols from /boot/kernel/iicbus.ko...Reading symbols from /boot/kernel/iicbus.ko.symbols...done. done. Loaded symbols for /boot/kernel/iicbus.ko Reading symbols from /boot/kernel/iic.ko...Reading symbols from /boot/kernel/iic.ko.symbols...done. done. Loaded symbols for /boot/kernel/iic.ko Reading symbols from /boot/kernel/drm.ko...Reading symbols from /boot/kernel/drm.ko.symbols...done. done. Loaded symbols for /boot/kernel/drm.ko Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from /boot/kernel/geom_eli.ko.symbols...done. done. Loaded symbols for /boot/kernel/geom_eli.ko Reading symbols from /boot/kernel/crypto.ko...Reading symbols from /boot/kernel/crypto.ko.symbols...done. done. Loaded symbols for /boot/kernel/crypto.ko Reading symbols from /boot/kernel/zlib.ko...Reading symbols from /boot/kernel/zlib.ko.symbols...done. done. Loaded symbols
Re: [panic] intr_event_execute_handlers() - Corrupted DWARF expression
On Wednesday, January 18, 2012 5:01:37 pm Glen Barber wrote: > Hi, > > I'm running -CURRENT from about 5 days ago: > > nucleus# uname -a > FreeBSD nucleus 10.0-CURRENT FreeBSD 10.0-CURRENT #3 r230037M: Fri Jan > 13 17:48:14 EST 2012 gjb@nucleus:/usr/obj/usr/src/sys/NUCLEUS amd64 > > (The 'M' is kib's DRM patches for Intel GPU.) > > So far, I haven't had much problem with this laptop, but just had the > machine panic. > > I have kgdb output attached, and I'll be happy to provide whatever > additional information that may be needed. > > I have core.txt.N available here: > > http://people.freebsd.org/~gjb/core.txt In kgdb, can you go to frame 6 and 'p td->td_lock'. If that is non-null, can you do 'p *td->td_lock'? -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
