Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On Thu, 27 Jun 2013 23:58:33 +0200 Kristof Provost kris...@sigsegv.be wrote: On 2013-06-24 22:08:01 (+0200), Alexander Leidinger alexan...@leidinger.net wrote: On Mon, 24 Jun 2013 12:15:18 +0200 Kristof Provost kris...@sigsegv.be wrote: For what it's worth, I'm running into exactly the same problem. (amd64, stable/9). I have no custom settings in /etc/make.conf or /etc/src.conf I had a short discussion with the maintainer of our stress-test-suite, he was able to create a test-case which triggers the problem. I've been bisecting for a little bit, and while I'm not 100% sure yet, there is one likely culprit right now: r249643. It's an MFC with a number of ZFS changes relating to a refactoring of the ioctl() interface. It is, unfortunately, also a rather large commit. Martin, the issue here is that starting a jail with a recent -current panics, if the jail has a dataset assigned to it during start (and the rc.d zfs scripts kicks in). At least in my case the jail contains an userland from before the change and the jail host a current userland. Any ideas / suggestions? pho@ has a test case for this. Bye, Alexander. -- http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On 2013-06-29 12:01, Alexander Leidinger wrote: On Thu, 27 Jun 2013 23:58:33 +0200 Kristof Provost kris...@sigsegv.be wrote: On 2013-06-24 22:08:01 (+0200), Alexander Leidinger alexan...@leidinger.net wrote: On Mon, 24 Jun 2013 12:15:18 +0200 Kristof Provost kris...@sigsegv.be wrote: For what it's worth, I'm running into exactly the same problem. (amd64, stable/9). I have no custom settings in /etc/make.conf or /etc/src.conf I had a short discussion with the maintainer of our stress-test-suite, he was able to create a test-case which triggers the problem. I've been bisecting for a little bit, and while I'm not 100% sure yet, there is one likely culprit right now: r249643. It's an MFC with a number of ZFS changes relating to a refactoring of the ioctl() interface. It is, unfortunately, also a rather large commit. Martin, the issue here is that starting a jail with a recent -current panics, if the jail has a dataset assigned to it during start (and the rc.d zfs scripts kicks in). At least in my case the jail contains an userland from before the change and the jail host a current userland. Any ideas / suggestions? pho@ has a test case for this. Bye, Alexander. Hi Alexander, some input would be great (at least the panic message - ideally from a debug kernel). Cheers, mm ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On Sat, 29 Jun 2013 14:02:35 +0200 Martin Matuska m...@freebsd.org wrote: some input would be great (at least the panic message - ideally from a debug kernel). The bt in the minidump is useless, but I made a bt directly in the kernel debugger: ---snip--- Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x0 fault code = supervisor read instruction, page not present instruction pointer = 0x20:0x0 stack pointer = 0x28:0xff839e79d930 frame pointer = 0x28:0xff839e79d9e0 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 2183 (zfs) db bt Tracing pid 2356 uart_sab82532_class() at 0 devfs_ioctl_f() at devfs_ioctl_f+0xf0 kern_ioctl() at kern_ioctl+0x1d7 sys_ioctl() at sys_ioctl+0x142 ---snip--- The test case is easy, create a dataset for a jail, add something like this to /etc/devfs.rules: ---snip--- [devfsrules_unhide_zfs=12] add path zfs unhide [devfsrules_jail_withzfs=17] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add include $devfsrules_unhide_zfs ---snip--- Attach the dataset to the jail and see the box panicing on the use of the zfs command in the jail (maybe you don't even need to attach the dataset to the jail, maybe just the above devfs stuff is enough). The default jail scripts don't attach a ZFS dataset to a jail, the corresponding ezjail-script code is # Attach ZFS-datasets to the jail for zfs in ${ezjail_zfs_datasets}; do /sbin/zfs jail ${ezjail_id} ${zfs} || echo -n Error: ${zfs} could not be configured done Bye, Alexander. -- http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
This was an obvious error by me - I forgot to register zfs_ioc_jail and zfs_ioc_unjail using the new functions. Amazing noone noticed this by now until it got merged down to stable/8. In addition, I see no need to log these operations to the zpool history as they cause no on-disk changes, so I have disabled logging for these calls. Please test the patch from current in r252380. http://svnweb.freebsd.org/base?view=revisionrevision=252380 On 29.6.2013 17:00, Alexander Leidinger wrote: On Sat, 29 Jun 2013 14:02:35 +0200 Martin Matuska m...@freebsd.org wrote: some input would be great (at least the panic message - ideally from a debug kernel). The bt in the minidump is useless, but I made a bt directly in the kernel debugger: ---snip--- Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x0 fault code = supervisor read instruction, page not present instruction pointer = 0x20:0x0 stack pointer = 0x28:0xff839e79d930 frame pointer = 0x28:0xff839e79d9e0 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 2183 (zfs) db bt Tracing pid 2356 uart_sab82532_class() at 0 devfs_ioctl_f() at devfs_ioctl_f+0xf0 kern_ioctl() at kern_ioctl+0x1d7 sys_ioctl() at sys_ioctl+0x142 ---snip--- The test case is easy, create a dataset for a jail, add something like this to /etc/devfs.rules: ---snip--- [devfsrules_unhide_zfs=12] add path zfs unhide [devfsrules_jail_withzfs=17] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add include $devfsrules_unhide_zfs ---snip--- Attach the dataset to the jail and see the box panicing on the use of the zfs command in the jail (maybe you don't even need to attach the dataset to the jail, maybe just the above devfs stuff is enough). The default jail scripts don't attach a ZFS dataset to a jail, the corresponding ezjail-script code is # Attach ZFS-datasets to the jail for zfs in ${ezjail_zfs_datasets}; do /sbin/zfs jail ${ezjail_id} ${zfs} || echo -n Error: ${zfs} could not be configured done Bye, Alexander. -- Martin Matuska FreeBSD committer http://blog.vx.sk ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On Sat, 29 Jun 2013 18:49:16 +0200 Martin Matuska m...@freebsd.org wrote: Please test the patch from current in r252380. Buildworld+kernel in progress, expect feedback soon. FYI: you misspelled my FreeBSD address in the commit. Bye, Alexander. -- http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On 2013-06-29 18:49:16 (+0200), Martin Matuska m...@freebsd.org wrote: This was an obvious error by me - I forgot to register zfs_ioc_jail and zfs_ioc_unjail using the new functions. Amazing noone noticed this by now until it got merged down to stable/8. In addition, I see no need to log these operations to the zpool history as they cause no on-disk changes, so I have disabled logging for these calls. Please test the patch from current in r252380. http://svnweb.freebsd.org/base?view=revisionrevision=252380 This fixes the panic for me (on stable/9). Thanks, Kristof ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On Sat, 29 Jun 2013 23:05:20 +0200 Kristof Provost kris...@sigsegv.be wrote: On 2013-06-29 18:49:16 (+0200), Martin Matuska m...@freebsd.org wrote: This was an obvious error by me - I forgot to register zfs_ioc_jail and zfs_ioc_unjail using the new functions. Amazing noone noticed this by now until it got merged down to stable/8. In addition, I see no need to log these operations to the zpool history as they cause no on-disk changes, so I have disabled logging for these calls. Please test the patch from current in r252380. http://svnweb.freebsd.org/base?view=revisionrevision=252380 This fixes the panic for me (on stable/9). I confirm for -current. Bye, Alexander. -- http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On 2013-06-24 22:08:01 (+0200), Alexander Leidinger alexan...@leidinger.net wrote: On Mon, 24 Jun 2013 12:15:18 +0200 Kristof Provost kris...@sigsegv.be wrote: For what it's worth, I'm running into exactly the same problem. (amd64, stable/9). I have no custom settings in /etc/make.conf or /etc/src.conf I had a short discussion with the maintainer of our stress-test-suite, he was able to create a test-case which triggers the problem. I've been bisecting for a little bit, and while I'm not 100% sure yet, there is one likely culprit right now: r249643. It's an MFC with a number of ZFS changes relating to a refactoring of the ioctl() interface. It is, unfortunately, also a rather large commit. Regards, Kristof ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On 2013-06-14 23:07:02 (+0200), Alexander Leidinger alexan...@leidinger.net wrote: The bt in the minidump is useless, but I made a bt directly in the kernel debugger: ---snip--- Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x0 fault code = supervisor read instruction, page not present instruction pointer = 0x20:0x0 stack pointer = 0x28:0xff839e79d930 frame pointer = 0x28:0xff839e79d9e0 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 2183 (zfs) db bt Tracing pid 2356 uart_sab82532_class() at 0 devfs_ioctl_f() at devfs_ioctl_f+0xf0 kern_ioctl() at kern_ioctl+0x1d7 sys_ioctl() at sys_ioctl+0x142 ---snip--- For what it's worth, I'm running into exactly the same problem. (amd64, stable/9). I have no custom settings in /etc/make.conf or /etc/src.conf Regards, Kristof ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On Mon, 24 Jun 2013 12:15:18 +0200 Kristof Provost kris...@sigsegv.be wrote: For what it's worth, I'm running into exactly the same problem. (amd64, stable/9). I have no custom settings in /etc/make.conf or /etc/src.conf I had a short discussion with the maintainer of our stress-test-suite, he was able to create a test-case which triggers the problem. Bye, Alexander. -- http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On Fri, Jun 14, 2013 at 11:07:02PM +0200, Alexander Leidinger wrote: db bt Tracing pid 2356 uart_sab82532_class() at 0 devfs_ioctl_f() at devfs_ioctl_f+0xf0 kern_ioctl() at kern_ioctl+0x1d7 sys_ioctl() at sys_ioctl+0x142 ---snip--- Anyone with a pointer to an explanation how to convert those pointers into source locations? kgdb l *devfs_ioctl_f+0xf0 -- Mikolaj Golub ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On Sat, 15 Jun 2013 11:17:07 +0300 Mikolaj Golub troc...@freebsd.org wrote: On Fri, Jun 14, 2013 at 11:07:02PM +0200, Alexander Leidinger wrote: db bt Tracing pid 2356 uart_sab82532_class() at 0 devfs_ioctl_f() at devfs_ioctl_f+0xf0 kern_ioctl() at kern_ioctl+0x1d7 sys_ioctl() at sys_ioctl+0x142 ---snip--- Anyone with a pointer to an explanation how to convert those pointers into source locations? kgdb l *devfs_ioctl_f+0xf0 I have the old kernel loaded and have the new one in kgdb. It seems it is loading the symbols of the modules for the old kernel. As devfs is not a module, it shouldn't matter here. ---snip--- (kgdb) l *devfs_ioctl_f+0xf0 0x80346dd0 is in devfs_ioctl_f (/space/system/usr_src/sys/fs/devfs/devfs_vnops.c:757). 752 error = copyout(p, fgn-buf, i); 753 td-td_fpop = fpop; 754 dev_relthread(dev, ref); 755 return (error); 756 } 757 error = dsw-d_ioctl(dev, com, data, fp-f_flag, td); 758 td-td_fpop = NULL; 759 dev_relthread(dev, ref); 760 if (error == ENOIOCTL) 761 error = ENOTTY; ---snip--- I would assume I can not print anything from there with my core-dump, as the backtrace is not usable. Bye, Alexander. -- http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On Wed, 12 Jun 2013 22:39:53 +0200 Dimitry Andric d...@freebsd.org wrote: On Jun 12, 2013, at 22:30, Alexander Leidinger alexan...@leidinger.net wrote: I try to update from a pre-clang world (r242511M) to now (r251618M). The resulting kernel boots, but while starting some jails (with ezjail from ports, so fairly late in the boot process) I get a kernel panic (IIRC zfs trying to access page 0). If you are running on i386, it might be a stack overflow? Try increasing the stack a little, it might help in that case. In any case, please try to get a backtrace. The bt in the minidump is useless, but I made a bt directly in the kernel debugger: ---snip--- Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x0 fault code = supervisor read instruction, page not present instruction pointer = 0x20:0x0 stack pointer = 0x28:0xff839e79d930 frame pointer = 0x28:0xff839e79d9e0 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 2183 (zfs) db bt Tracing pid 2356 uart_sab82532_class() at 0 devfs_ioctl_f() at devfs_ioctl_f+0xf0 kern_ioctl() at kern_ioctl+0x1d7 sys_ioctl() at sys_ioctl+0x142 ---snip--- Anyone with a pointer to an explanation how to convert those pointers into source locations? The minidump is available at http://www.leidinger.net/test/core.txt.1.bz2 unfortunately the bt in there is crap. Bye, Alexander. -- http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On Wed, 12 Jun 2013 15:10:59 -0700 Peter Wemm pe...@wemm.org wrote: On Wed, Jun 12, 2013 at 1:39 PM, Dimitry Andric d...@freebsd.org wrote: If you are running on i386, it might be a stack overflow? Try increasing the stack a little, it might help in that case. For i386 I'd be more inclined to suspect KVA exhaustion. This is just a shot in the dark. If this is amd64, then never mind, KVA_PAGES is meaningless there. Sorry, I forgot to specify, it's amd64. Bye, Alexander. -- http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
Hi, I try to update from a pre-clang world (r242511M) to now (r251618M). The resulting kernel boots, but while starting some jails (with ezjail from ports, so fairly late in the boot process) I get a kernel panic (IIRC zfs trying to access page 0). Before I try to get some time to debug this, I would like to know if there are some known incompatibilities with my make.conf settings. With gcc I used successfully this: ---snip--- COPTFLAGS= -O2 -pipe CPUTYPE?=core2 ---snip--- With those settings I first did a buildworld, then a buildkernel with the r251618 sources. Are there some known issues with those settings? If yes, any suggestions what I should use instead? If not, would it be beneficial to try with different settings (which ones)? Bye, Alexander. -- http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On Jun 12, 2013, at 22:30, Alexander Leidinger alexan...@leidinger.net wrote: I try to update from a pre-clang world (r242511M) to now (r251618M). The resulting kernel boots, but while starting some jails (with ezjail from ports, so fairly late in the boot process) I get a kernel panic (IIRC zfs trying to access page 0). If you are running on i386, it might be a stack overflow? Try increasing the stack a little, it might help in that case. In any case, please try to get a backtrace. Before I try to get some time to debug this, I would like to know if there are some known incompatibilities with my make.conf settings. With gcc I used successfully this: ---snip--- COPTFLAGS= -O2 -pipe CPUTYPE?=core2 ---snip--- With those settings I first did a buildworld, then a buildkernel with the r251618 sources. Are there some known issues with those settings? If yes, any suggestions what I should use instead? If not, would it be beneficial to try with different settings (which ones)? -O2 -pipe is the default setting, so it should work. I personally also always use CPUTYPE core2, and I have never seen any panics. Then again, I do not usually use jails intensively... -Dimitry ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On Wed, Jun 12, 2013 at 1:39 PM, Dimitry Andric d...@freebsd.org wrote: On Jun 12, 2013, at 22:30, Alexander Leidinger alexan...@leidinger.net wrote: I try to update from a pre-clang world (r242511M) to now (r251618M). The resulting kernel boots, but while starting some jails (with ezjail from ports, so fairly late in the boot process) I get a kernel panic (IIRC zfs trying to access page 0). If you are running on i386, it might be a stack overflow? Try increasing the stack a little, it might help in that case. For i386 I'd be more inclined to suspect KVA exhaustion. For non-PAE, as a shot in the dark, increase options KVA_PAGES=384 .. the default is 256 for PAE. that increases kernel KVA from 1GB to 1.5GB. For a PAE system, this number is multipled by 2, so a corresponding change is 512 - 768. This is just a shot in the dark. If this is amd64, then never mind, KVA_PAGES is meaningless there. -- Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV On IRC, talking about C++: BigKnife I think that it is a good thing I will never meet Bjarne on a street BigKnife cause really, I don't want to end up in prison or anything ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: zfs kernel panic, known incompatibilities with clang CPUTYPE/COPTFLAGS?
On 6/12/13 3:10 PM, Peter Wemm wrote: On Wed, Jun 12, 2013 at 1:39 PM, Dimitry Andric d...@freebsd.org wrote: On Jun 12, 2013, at 22:30, Alexander Leidinger alexan...@leidinger.net wrote: I try to update from a pre-clang world (r242511M) to now (r251618M). The resulting kernel boots, but while starting some jails (with ezjail from ports, so fairly late in the boot process) I get a kernel panic (IIRC zfs trying to access page 0). If you are running on i386, it might be a stack overflow? Try increasing the stack a little, it might help in that case. For i386 I'd be more inclined to suspect KVA exhaustion. For non-PAE, as a shot in the dark, increase options KVA_PAGES=384 .. the default is 256 for PAE. that increases kernel KVA from 1GB to 1.5GB. For a PAE system, this number is multipled by 2, so a corresponding change is 512 - 768. This is just a shot in the dark. If this is amd64, then never mind, KVA_PAGES is meaningless there. Is there some way we can get a pps ratelimited (or even one-time) message when the kva is almost exhausted? Could that help people? -- Alfred Perlstein VP Software Engineering, iXsystems ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org