[CURRENT]: weird memory/linker problem?
Hello. I face a strange problem on a set of CURRENT driven boxes. The systems in question are all the same version of CURRENT (more or less, a week or so discrepancy). The boxes affected have 8 GB of RAM and are old-style Core2Duo systems. The phenomenon: Starting up the box shows the operating system working. But sometimes it is impossible to start certain applications, like Firefox - they segfault. More disturbing is the fail of the linker when building world. Sometimes I get strange messages like relocation truncated to fit: R_X86_64_PC32 against symbol `__error' defined in .text when compiling/linking. The funny thing is: rebooting the box and doing exactly the same very often leaves the system then operable - starting applications works, compiling works! First I thought this could be a indication of a dying system and so I checked the memory for two days non-stop without any indication of anything wrong. The boxes do not have ECC RAM - it's Intel. I see this problem on two C2D based boxes relatively often (one E8400 two core, another Q6600 quadcore, both systems have 8 GB RAM). This phenomenon also occured two or three months ago on another machine with 32 GB RAM and a Core-i7 3930K, but it went away (it was the very same error as shown above). Another system, a i3-3220 with 16 GB RAM never showed the problem although that system build world also on a regular basis very frequent as the C2D systems do. Well, I feel a bit confused. On the first view, the problem looks weird and it indicates a kind of memory problem - but testing the memory didn't show anything wrong. Today windowmaker stopped starting due to a malformed command in one of windowmaker's library. I did reboot the box and everything was all right. Then, also today, I tried compiling world and I got a weird error message about a misspelled Int__xxx, I can not remember exactly the text, I rebooted and everything was all right again. Those errors are frequent on 8GB, C2D based systems and at the moment not present any more on more modern systems with more memory as described above. This could be a coincidence, but it is strange anyway. I do not exclude dying hardware, but I'd like to ask whether there is something strange going on with FreeBSD's memory management at the moment and whether those problems could also be triggered by some nasty bug? I never see a crash (which would also indicated faulty hardware), I mostly realise those strange behaviour either after a fresh boot or after I ran some memory disk i/o intensive jobs, like updating the ports tree. By the way, FreeBSD CURRENT suffer from a tremendous performance cut these days when compiling world and updating the ports tree and running portmaster. On one box, on which ports reside on a UFS partion, it takes more than 8 minutes to pass the portmaster -da, which is quick when not compiling world. On another system on which /usr/ports is residing on ZFS (the box has 16GB RAM!), it takes sometimes 30(!) minutes to perform a svn update while compiling world (that is the i3-3220 with 16 GB RAM system), it takes 6 - 15 minutes when the box is relaxed and updating the ports tree the first time (every subsequent update is much faster). Well, I know these reports of mine are a bit weird since I have no exact log of the problems, but I think if there is an issue not with the hardware, I report those in. Regards, oh signature.asc Description: PGP signature
Re: [CURRENT]: weird memory/linker problem?
When they segfault, where do they segfault? -a On 22 June 2014 07:56, O. Hartmann ohart...@zedat.fu-berlin.de wrote: Hello. I face a strange problem on a set of CURRENT driven boxes. The systems in question are all the same version of CURRENT (more or less, a week or so discrepancy). The boxes affected have 8 GB of RAM and are old-style Core2Duo systems. The phenomenon: Starting up the box shows the operating system working. But sometimes it is impossible to start certain applications, like Firefox - they segfault. More disturbing is the fail of the linker when building world. Sometimes I get strange messages like relocation truncated to fit: R_X86_64_PC32 against symbol `__error' defined in .text when compiling/linking. The funny thing is: rebooting the box and doing exactly the same very often leaves the system then operable - starting applications works, compiling works! First I thought this could be a indication of a dying system and so I checked the memory for two days non-stop without any indication of anything wrong. The boxes do not have ECC RAM - it's Intel. I see this problem on two C2D based boxes relatively often (one E8400 two core, another Q6600 quadcore, both systems have 8 GB RAM). This phenomenon also occured two or three months ago on another machine with 32 GB RAM and a Core-i7 3930K, but it went away (it was the very same error as shown above). Another system, a i3-3220 with 16 GB RAM never showed the problem although that system build world also on a regular basis very frequent as the C2D systems do. Well, I feel a bit confused. On the first view, the problem looks weird and it indicates a kind of memory problem - but testing the memory didn't show anything wrong. Today windowmaker stopped starting due to a malformed command in one of windowmaker's library. I did reboot the box and everything was all right. Then, also today, I tried compiling world and I got a weird error message about a misspelled Int__xxx, I can not remember exactly the text, I rebooted and everything was all right again. Those errors are frequent on 8GB, C2D based systems and at the moment not present any more on more modern systems with more memory as described above. This could be a coincidence, but it is strange anyway. I do not exclude dying hardware, but I'd like to ask whether there is something strange going on with FreeBSD's memory management at the moment and whether those problems could also be triggered by some nasty bug? I never see a crash (which would also indicated faulty hardware), I mostly realise those strange behaviour either after a fresh boot or after I ran some memory disk i/o intensive jobs, like updating the ports tree. By the way, FreeBSD CURRENT suffer from a tremendous performance cut these days when compiling world and updating the ports tree and running portmaster. On one box, on which ports reside on a UFS partion, it takes more than 8 minutes to pass the portmaster -da, which is quick when not compiling world. On another system on which /usr/ports is residing on ZFS (the box has 16GB RAM!), it takes sometimes 30(!) minutes to perform a svn update while compiling world (that is the i3-3220 with 16 GB RAM system), it takes 6 - 15 minutes when the box is relaxed and updating the ports tree the first time (every subsequent update is much faster). Well, I know these reports of mine are a bit weird since I have no exact log of the problems, but I think if there is an issue not with the hardware, I report those in. Regards, oh ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [CURRENT]: weird memory/linker problem?
On 2014-06-22 10:56, O. Hartmann wrote: Hello. I face a strange problem on a set of CURRENT driven boxes. The systems in question are all the same version of CURRENT (more or less, a week or so discrepancy). The boxes affected have 8 GB of RAM and are old-style Core2Duo systems. The phenomenon: Starting up the box shows the operating system working. But sometimes it is impossible to start certain applications, like Firefox - they segfault. More disturbing is the fail of the linker when building world. Sometimes I get strange messages like relocation truncated to fit: R_X86_64_PC32 against symbol `__error' defined in .text when compiling/linking. The funny thing is: rebooting the box and doing exactly the same very often leaves the system then operable - starting applications works, compiling works! First I thought this could be a indication of a dying system and so I checked the memory for two days non-stop without any indication of anything wrong. The boxes do not have ECC RAM - it's Intel. I see this problem on two C2D based boxes relatively often (one E8400 two core, another Q6600 quadcore, both systems have 8 GB RAM). This phenomenon also occured two or three months ago on another machine with 32 GB RAM and a Core-i7 3930K, but it went away (it was the very same error as shown above). Another system, a i3-3220 with 16 GB RAM never showed the problem although that system build world also on a regular basis very frequent as the C2D systems do. Well, I feel a bit confused. On the first view, the problem looks weird and it indicates a kind of memory problem - but testing the memory didn't show anything wrong. Today windowmaker stopped starting due to a malformed command in one of windowmaker's library. I did reboot the box and everything was all right. Then, also today, I tried compiling world and I got a weird error message about a misspelled Int__xxx, I can not remember exactly the text, I rebooted and everything was all right again. Those errors are frequent on 8GB, C2D based systems and at the moment not present any more on more modern systems with more memory as described above. This could be a coincidence, but it is strange anyway. I do not exclude dying hardware, but I'd like to ask whether there is something strange going on with FreeBSD's memory management at the moment and whether those problems could also be triggered by some nasty bug? I never see a crash (which would also indicated faulty hardware), I mostly realise those strange behaviour either after a fresh boot or after I ran some memory disk i/o intensive jobs, like updating the ports tree. By the way, FreeBSD CURRENT suffer from a tremendous performance cut these days when compiling world and updating the ports tree and running portmaster. On one box, on which ports reside on a UFS partion, it takes more than 8 minutes to pass the portmaster -da, which is quick when not compiling world. On another system on which /usr/ports is residing on ZFS (the box has 16GB RAM!), it takes sometimes 30(!) minutes to perform a svn update while compiling world (that is the i3-3220 with 16 GB RAM system), it takes 6 - 15 minutes when the box is relaxed and updating the ports tree the first time (every subsequent update is much faster). Well, I know these reports of mine are a bit weird since I have no exact log of the problems, but I think if there is an issue not with the hardware, I report those in. Regards, oh In order to get a better benchmark for 'svn update' on the ports tree if you 'zfs unmount pool/usr/ports' it will flush all ARC entries for that dataset, then 'zfs mount pool/usr/ports' and run the test again. This should give you more reproducible results -- Allan Jude signature.asc Description: OpenPGP digital signature
Re: ucom_free Fatal trap on shutdown / module unload
Hi I tried replacing DRIVER_MODULE(uhso, uhub, uhso_driver, uhso_devclass, uhso_driver_loaded, 0); with DRIVER_MODULE_ORDERED(uhso, uhub, uhso_driver, uhso_devclass, uhso_driver_loaded, 0, SI_ORDER_ANY); but makes no difference.. Don't know if its relevant but with ucom debug on I get a message just before crash in method ucom_close that it tries to close a connection that has already been closed. -- Johannes Lundberg BRILLIANTSERVICE CO., LTD. On Fri, Jun 20, 2014 at 4:51 PM, Hans Petter Selasky h...@selasky.org wrote: On 06/20/14 04:25, Lundberg, Johannes wrote: Hi I'm getting this error on 11-CURRENT amd64 (snapshot from June). (see attached image) Worked fine with 10 I think.. The ucom module is loaded as a dependency by the uhso module. Any clues? -- Johannes Lundberg [image: Inline image 1] Hi, Possibly something similar to what was done in USB audio that DRIVER_MODULE_ORDERED() needs to be used instead of DRIVER_MODULE(). --HPS -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 秘密保持について：この電子メールは、名宛人に送信したものであり、秘匿特権の対象となる情報を含んでいます。 もし、名宛人以外の方が受信された場合、このメールの破棄、およびこのメールに関する一切の開示、 複写、配布、その他の利用、または記載内容に基づくいかなる行動もされないようお願い申し上げます。 --- CONFIDENTIALITY NOTE: The information in this email is confidential and intended solely for the addressee. Disclosure, copying, distribution or any other action of use of this email by person other than intended recipient, is prohibited. If you are not the intended recipient and have received this email in error, please destroy the original message. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Latest -current panic in uaudio_detach() / bus_dmamem_free()
Always happens at shutdown after all buffers are synced, see screenshot: http://i.imgur.com/8WXTMPj.png -- http://ache.vniz.net/ ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Latest -current panic in uaudio_detach() / bus_dmamem_free()
On Mon, 23 Jun 2014 06:04:20 +0400
Andrey Chernov a...@freebsd.org wrote:
Always happens at shutdown after all buffers are synced, see
screenshot: http://i.imgur.com/8WXTMPj.png
--
http://ache.vniz.net/
Hi Andrey,
there's not to much to go on from the screenshoot alone and one would
expect more details on the crash from people with your experience :)
Please provide us with the information on the actual audio hardware
you are using, preferably in form of a dmesg output. This revision is
your culpit:
http://svnweb.freebsd.org/changeset/base/267581 and I have strong
suspicion that restoring the NULL check on dmatag in the chunk below
will cure your crash.
-- Modified: head/sys/dev/sound/pcm/buffer.c
==
--- head/sys/dev/sound/pcm/buffer.c Tue Jun 17 14:47:49
2014(r267580) +++ head/sys/dev/sound/pcm/buffer.c Tue
Jun 17 16:07:57 2014(r267581) @@ -139,10 +139,9 @@
sndbuf_free(struct snd_dbuf *b)
if (b-buf) {
if (b-flags SNDBUF_F_MANAGED) {
- if (b-dmamap)
+ if (b-buf_addr)
bus_dmamap_unload(b-dmatag,
b-dmamap);
- if (b-dmatag)
- bus_dmamem_free(b-dmatag, b-buf,
b-dmamap);
+ bus_dmamem_free(b-dmatag, b-buf, b-dmamap);
} else
free(b-buf, M_DEVBUF);
}
--
Alexander Kabaev
signature.asc
Description: PGP signature
Re: ucom_free Fatal trap on shutdown / module unload
On 06/23/14 03:30, Lundberg, Johannes wrote:
Hi
I tried replacing
DRIVER_MODULE(uhso, uhub, uhso_driver, uhso_devclass, uhso_driver_loaded,
0);
with
DRIVER_MODULE_ORDERED(uhso, uhub, uhso_driver, uhso_devclass,
uhso_driver_loaded, 0, SI_ORDER_ANY);
but makes no difference..
Don't know if its relevant but with ucom debug on I get a message just
before crash in method ucom_close that it tries to close a connection that
has already been closed.
Hi Johannes,
Try the opposite:
DRIVER_MODULE_ORDERED(uhso, uhub, uhso_driver, uhso_devclass,
uhso_driver_loaded, 0, SI_ORDER_MIDDLE + 1);
Because I suspect that the uhso_ifnet_unit unrhdr is freed before the
fake detach is executed:
static int
uhso_driver_loaded(struct module *mod, int what, void *arg)
{
switch (what) {
case MOD_LOAD:
/* register our autoinstall handler */
uhso_etag = EVENTHANDLER_REGISTER(usb_dev_configured,
uhso_test_autoinst, NULL, EVENTHANDLER_PRI_ANY);
/* create our unit allocator for inet devs */
uhso_ifnet_unit = new_unrhdr(0, INT_MAX, NULL);
break;
case MOD_UNLOAD:
EVENTHANDLER_DEREGISTER(usb_dev_configured, uhso_etag);
delete_unrhdr(uhso_ifnet_unit);
break;
default:
return (EOPNOTSUPP);
}
return (0);
}
Alternativly set uhso_ifnet_unit to NULL and check this in probe and
attach!
--HPS
--HPS
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Latest -current panic in uaudio_detach() / bus_dmamem_free()
On 06/23/14 04:46, Alexander Kabaev wrote: On Mon, 23 Jun 2014 06:04:20 +0400 Andrey Chernov a...@freebsd.org wrote: Always happens at shutdown after all buffers are synced, see screenshot: http://i.imgur.com/8WXTMPj.png -- http://ache.vniz.net/ Hi Andrey, there's not to much to go on from the screenshoot alone and one would expect more details on the crash from people with your experience :) Please provide us with the information on the actual audio hardware you are using, preferably in form of a dmesg output. This revision is your culpit: http://svnweb.freebsd.org/changeset/base/267581 and I have strong suspicion that restoring the NULL check on dmatag in the chunk below will cure your crash. Backtrace here: usbconfig -d 0.4 reset uaudio0: at uhub1, port 2, addr 4 (disconnected) vm_fault(0xc0661400, 0, 1, 0) - 1 Fatal kernel mode data abort: 'Translation Fault (P)' trapframe: 0xd28b8b58 FSR=0017, FAR=002c, spsr=6113 r0 =, r1 =c1b35000, r2 =, r3 = r4 =c1a24000, r5 =, r6 =c1b3338c, r7 =c172e150 r8 =c1b35000, r9 =, r10=c162a400, r11=d28b8bd0 r12=c1bc9ad4, ssp=d28b8ba8, slr=c1b9855c, pc =c048fa3c [ thread pid 14 tid 100037 ] Stopped at bus_dmamem_free+0x10: ldr r0, [r9, #0x02c] db bt Tracing pid 14 tid 100037 td 0xc1712960 db_trace_self() at db_trace_self pc = 0xc0492958 lr = 0xc0130f38 (db_stack_trace+0xf4) sp = 0xd28b8860 fp = 0xd28b8878 r10 = 0xc0660180 db_stack_trace() at db_stack_trace+0xf4 pc = 0xc0130f38 lr = 0xc01308a8 (db_command+0x270) sp = 0xd28b8880 fp = 0xd28b8920 r4 = 0x r5 = 0x r6 = 0x db_command() at db_command+0x270 pc = 0xc01308a8 lr = 0xc013060c (db_command_loop+0x60) sp = 0xd28b8928 fp = 0xd28b8938 r4 = 0xc04d2192 r5 = 0xc04ec76c r6 = 0xc066016c r7 = 0xc058b540 r8 = 0xc0656294 r9 = 0xc0656290 r10 = 0x0001 db_command_loop() at db_command_loop+0x60 pc = 0xc013060c lr = 0xc0132fd4 (db_trap+0xd8) sp = 0xd28b8940 fp = 0xd28b8a60 r4 = 0x r5 = 0xc0660178 r6 = 0xc06562c0 db_trap() at db_trap+0xd8 pc = 0xc0132fd4 lr = 0xc028efbc (kdb_trap+0xbc) sp = 0xd28b8a68 fp = 0xd28b8a88 r4 = 0x r5 = 0x0017 r6 = 0xc06562c0 r7 = 0xc058b540 kdb_trap() at kdb_trap+0xbc pc = 0xc028efbc lr = 0xc04a5194 (dab_fatal+0x174) sp = 0xd28b8a90 fp = 0xd28b8aa8 r4 = 0xd28b8b58 r5 = 0x0017 r6 = 0x61d3 r7 = 0x002c r8 = 0xd28b8b58 r9 = 0x0013 r10 = 0x0001 dab_fatal() at dab_fatal+0x174 pc = 0xc04a5194 lr = 0xc04a4f4c (data_abort_handler+0x3e8) sp = 0xd28b8ab0 fp = 0xd28b8b50 r4 = 0xc16be3cc r5 = 0xc1712960 r6 = 0xd28b8eb0 r7 = 0x data_abort_handler() at data_abort_handler+0x3e8 pc = 0xc04a4f4c lr = 0xc04944d4 (exception_exit) sp = 0xd28b8b58 fp = 0xd28b8bd0 r4 = 0xc1a24000 r5 = 0x r6 = 0xc1b3338c r7 = 0xc172e150 r8 = 0xc1b35000 r9 = 0x r10 = 0xc162a400 exception_exit() at exception_exit pc = 0xc04944d4 lr = 0xc1b9855c (sndbuf_free+0x80) sp = 0xd28b8ba8 fp = 0xd28b8bd0 r0 = 0x r1 = 0xc1b35000 r2 = 0x r3 = 0x r4 = 0xc1a24000 r5 = 0x r6 = 0xc1b3338c r7 = 0xc172e150 r8 = 0xc1b35000 r9 = 0x r10 = 0xc162a400 r12 = 0xc1bc9ad4 bus_dmamem_free() at bus_dmamem_free+0x10 pc = 0xc048fa3c lr = 0xc1b984c4 (sndbuf_destroy+0x14) sp = 0xd28b8bd8 fp = 0xd28b8be0 r4 = 0xc162ae00 r5 = 0xc1a24000 r6 = 0xd28b8bd0 r7 = 0xc1b9855c r8 = 0x r9 = 0xc1a24000 Unknown entry: 0 sndbuf_destroy() at sndbuf_destroy+0x14 pc = 0xc1b984c4 lr = 0xc1b984c4 (sndbuf_destroy+0x14) sp = 0xd28b8bd8 fp = 0xd28b8be0 Unable to unwind into user mode Please fix ASAP. Should be trivial to reproduce. Possibly a double free. In case of USB audio sndbuf_destroy() should not free any bus dmamem or know about busdma, because all of this is done by the USB stack! --HPS ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Latest -current panic in uaudio_detach() / bus_dmamem_free()
On 23.06.2014 6:46, Alexander Kabaev wrote:
Please provide us with the information on the actual audio hardware
you are using, preferably in form of a dmesg output.
uaudio0: vendor 0x046d product 0x0990, class 239/2, rev 2.00/0.05, addr 7 on
usbus3
uaudio0: No playback.
uaudio0: Record: 16000 Hz, 1 ch, 16-bit S-LE PCM format, 2x8ms buffer.
uaudio0: No MIDI sequencer.
pcm7: USB audio on uaudio0
uaudio0: No HID volume keys found.
Thanx, after backing out the patch below this panic is gone. Probably even
additional b-dmatag NULL check is needed for bus_dmamap_unload() too.
This revision is
your culpit:
http://svnweb.freebsd.org/changeset/base/267581 and I have strong
suspicion that restoring the NULL check on dmatag in the chunk below
will cure your crash.
-- Modified: head/sys/dev/sound/pcm/buffer.c
==
--- head/sys/dev/sound/pcm/buffer.c Tue Jun 17 14:47:49
2014 (r267580) +++ head/sys/dev/sound/pcm/buffer.c Tue
Jun 17 16:07:57 2014 (r267581) @@ -139,10 +139,9 @@
sndbuf_free(struct snd_dbuf *b)
if (b-buf) {
if (b-flags SNDBUF_F_MANAGED) {
- if (b-dmamap)
+ if (b-buf_addr)
bus_dmamap_unload(b-dmatag,
b-dmamap);
- if (b-dmatag)
- bus_dmamem_free(b-dmatag, b-buf,
b-dmamap);
+ bus_dmamem_free(b-dmatag, b-buf, b-dmamap);
} else
free(b-buf, M_DEVBUF);
}
--
http://ache.vniz.net/
signature.asc
Description: OpenPGP digital signature
Re: ucom_free Fatal trap on shutdown / module unload
I added some logging to see what is going on and this is what I got (none
of the proposed solution solved the problem)
uhso_detach gets called 7 times (for oid 0-6). It crashes the 2nd time on
the call usbd_transfer_unsetup(sc-sc_xfer, 3);
--
Johannes Lundberg
BRILLIANTSERVICE CO., LTD.
On Mon, Jun 23, 2014 at 11:55 AM, Hans Petter Selasky h...@selasky.org
wrote:
On 06/23/14 03:30, Lundberg, Johannes wrote:
Hi
I tried replacing
DRIVER_MODULE(uhso, uhub, uhso_driver, uhso_devclass, uhso_driver_loaded,
0);
with
DRIVER_MODULE_ORDERED(uhso, uhub, uhso_driver, uhso_devclass,
uhso_driver_loaded, 0, SI_ORDER_ANY);
but makes no difference..
Don't know if its relevant but with ucom debug on I get a message just
before crash in method ucom_close that it tries to close a connection that
has already been closed.
Hi Johannes,
Try the opposite:
DRIVER_MODULE_ORDERED(uhso, uhub, uhso_driver, uhso_devclass,
uhso_driver_loaded, 0, SI_ORDER_MIDDLE + 1);
Because I suspect that the uhso_ifnet_unit unrhdr is freed before the
fake detach is executed:
static int
uhso_driver_loaded(struct module *mod, int what, void *arg)
{
switch (what) {
case MOD_LOAD:
/* register our autoinstall handler */
uhso_etag = EVENTHANDLER_REGISTER(usb_dev_configured,
uhso_test_autoinst, NULL, EVENTHANDLER_PRI_ANY);
/* create our unit allocator for inet devs */
uhso_ifnet_unit = new_unrhdr(0, INT_MAX, NULL);
break;
case MOD_UNLOAD:
EVENTHANDLER_DEREGISTER(usb_dev_configured, uhso_etag);
delete_unrhdr(uhso_ifnet_unit);
break;
default:
return (EOPNOTSUPP);
}
return (0);
}
Alternativly set uhso_ifnet_unit to NULL and check this in probe and
attach!
--HPS
--HPS
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
秘密保持について：この電子メールは、名宛人に送信したものであり、秘匿特権の対象となる情報を含んでいます。
もし、名宛人以外の方が受信された場合、このメールの破棄、およびこのメールに関する一切の開示、
複写、配布、その他の利用、または記載内容に基づくいかなる行動もされないようお願い申し上げます。
---
CONFIDENTIALITY NOTE: The information in this email is confidential
and intended solely for the addressee.
Disclosure, copying, distribution or any other action of use of this
email by person other than intended recipient, is prohibited.
If you are not the intended recipient and have received this email in
error, please destroy the original message.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
