Re: amrd disk performance drop after running under high load
Alexey Popov wrote: After some time of running under high load disk performance become expremely poor. At that periods 'systat -vm 1' shows something like this: What does "high load" mean? You need to explain the system workload more. Disks amrd0 KB/t 85.39 tps 5 MB/s 0.38 % busy 99 Apart of all, I tried to make mutex profiling and here's the results (sorted by the total number of acquisitions): Bad case: 102 223514 273977 0 14689 1651568 /usr/src/sys/vm/uma_core.c:2349 (512) 950 263099 273968 0 15004 14427 /usr/src/sys/vm/uma_core.c:2450 (512) 108 150422 175840 0 10978 22988519 /usr/src/sys/vm/uma_core.c:1888 (mbuf) > Here you can see that high UMA activity happens in periods of low disk > performance. But I'm not sure whether this is a root of the problem, not > a consequence. The extremely high contention there does seem to say you have a mbuf starvation problem and not a disk problem. I don't know why this would be happening off-hand. Can you also provide more details about the system hardware and configuration? Kris ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 6.2: reproducible hang on amd64, traced to 24h of commits
fwiw, i have not traced it down to a commit (got fed up with hangs), but conclusively singled out smartmontools as the trigger. after adding 2 more disks, machine wouldn't even boot up past starting smartmontools, locking up hard with the same symptoms. with smartmontools disabled, it booted up and has been up for > 2 months now. Deomid Ryabkov wrote: ok, now that the machine has been up for 10 days, i am reasonably sure i've close enough to this one. back in january i cvsupped to -STABLE and the box (dual head opteron box) started hanging. and i mean it dies completely. i have all debug options and a working serial console, but still it just dies and both serial and system console are unresponsive. no panic message on either, nothing. pretty sad. the kernel config is vanilla SMP GENERIC, with all debug options i could think of enabled (after it started hanging). so the first thing i did after rebooting the box a couple of times is fall back to kernel.old (6.1-STABLE circa august '06). no hangs. i then started incrementally updating, gradually getting closer to jan 22. long story short, i seem to have isolated the problem to commits made between date=2006.12.28.00.00.00 and date=2006.12.29.00.00.00. last hang i had was when running the 12/29 kernel, now it's 12/28 and the box has been up for 2 weeks already. based on previois experience i'm pretty certain that this is it. with bad kernel the box would never stay up more than a few days, never more than 5. between 12/28 and 12/29 i see some changes to /sys/amd64/ and /sys/pci/, which might've be the cause. i will probably start looking into individual changes, but if anyone more experienced than me could take a look, it'd be appreciated. i am willing to try patches. i confirmed that recent (as of 3 weeks or so) -STABLE still has this problem. thanks in advance. files under /sys that were changed between 12/28 and 12/29: Edit src/sys/amd64/amd64/mptable_pci.c Edit src/sys/amd64/pci/pci_bus.c Edit src/sys/contrib/dev/ath/public/wackelf.c Edit src/sys/dev/acpica/acpi_pci.c Edit src/sys/dev/acpica/acpi_pcib_acpi.c Edit src/sys/dev/acpica/acpi_pcib_pci.c Checkout src/sys/dev/ath/if_ath.c Edit src/sys/dev/cardbus/cardbus.c Edit src/sys/dev/drm/drm_agpsupport.c Edit src/sys/dev/pci/pci.c Edit src/sys/dev/pci/pci_if.m Edit src/sys/dev/pci/pci_pci.c Edit src/sys/dev/pci/pci_private.h Edit src/sys/dev/pci/pcib_private.h Edit src/sys/dev/pci/pcivar.h Edit src/sys/i386/i386/mptable_pci.c Edit src/sys/i386/pci/pci_bus.c Edit src/sys/kern/subr_bus.c Checkout src/sys/netgraph/ng_deflate.h Edit src/sys/pci/agp.c Edit src/sys/pci/agpreg.h Edit src/sys/powerpc/ofw/ofw_pcib_pci.c Edit src/sys/sparc64/pci/apb.c Edit src/sys/sparc64/pci/ofw_pcib.c Edit src/sys/sparc64/pci/ofw_pcibus.c Edit src/sys/sys/param.h kernel configuration used: include GENERIC options SMP options KDB options DDB makeoptions DEBUG=-g options INVARIANTS options INVARIANT_SUPPORT options WITNESS options DEBUG_LOCKS options DEBUG_VFS_LOCKS options DIAGNOSTIC -- Deomid Ryabkov aka Rojer [EMAIL PROTECTED] [EMAIL PROTECTED] ICQ: 8025844 smime.p7s Description: S/MIME Cryptographic Signature
Re: Useful tools missing from /rescue
On Sat, Oct 13, 2007 at 10:01:39AM +0400, Yar Tikhiy wrote: > On Wed, Oct 03, 2007 at 07:23:44PM -0700, David O'Brien wrote: > > I also don't see the need for pgrep - I think needing that says your > > system is running multiuser pretty well. > > First of all, I'd like to point out that /rescue doesn't need to > be as minimal as /stand used to. Now, /rescue is a compact yet > versatile set of essential tools that can help in any difficult > situation when /*bin:/usr/*bin are unusable for some reason, not > only in restoring a broken system while in single-user mode. A .. > As for pgrep+pkill, it can come handy if one has screwed up his > live system and wants to recover it without dropping the system to > single-user. But if we take this just a little bit farther then why don't we go back to a static /[s]bin except for the few things one might need LDAP, etc.. for? That is, what's the purpose in continuing to duplicate /[s]bin into /rescue? /rescue should be just enough to reasonably get a system who's shared libs are messed up working again. /stand was a left-over from installation and not intended to be a sysadmins' savor - it just happened to be because we didn't clean up / after the bits were laid down. > A valid objection to this point is that pgrep's job > can be done with a combination of ps(1) and sed(1), so it's just a > matter of convenience. I guess I'm still having trouble understanding why one would need 'ps' to fix a shared libs issue. Now is a reason to keep adding stuff to /rescue. Also why one would be running 'ps -aux', which is the only way I can think of to get more than one screen of output if a system is in trouble. > The price for it in terms of disk space is next to nothing, and there > are quite useless space hogs in /rescue already (see below on > /rescue/vi.) Considering how few people are skilled in ed(1) these days, we have little choice but include vi. > I won't speak for everyone, but I really like to use fancy shell > commands, particularly during hard times: loops, pipelines, etc. > So I don't have to enter many commands for a single task or browse I guess I'm not creative enough in the ways I've screwed up my systems and needed tools from /rescue. 8-) > > I don't see the purpose of chown - if you have to fall back to /rescue > > you're user 'root' - and you're trying to fix enough so you can use > > standard /*lib & /*bin .. > Having /rescue/chown is just a matter of completeness of the ch* > subset of /rescue tools because chown's job can't be done by any > other stock tools. If /rescue is complete enough, one can find > more applications for it. E.g., the loader, a kernel, and /rescue /rescue wasn't intended to be well orthogonal. /rescue was part of he corner stone of the deal to switch to shared /[s]bin. -- -- David ([EMAIL PROTECTED]) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Interrupt/speed problems with 6.2 NFS server
Hi, I have an new NFS server that is processing roughly 15mbit of NFS traffic that we recently upgraded from an older 4.10 box. It has a 3-ware raid card, and is serving NFS out a single em nic to LAN clients. The machine works great just serving NFS, but when I try to copy data from one raid volume to another for backups, the machine's NFS performance goes way down, and NFSops start taking multiple seconds to perform. The file copy goes quite quickly, as would be expected. The console of the machine also starts to lag pretty badly, and I get the 'typing through mud' effect. I use rdist6 to do the backup. My first impression was that I was having interrupt issues, since during the backup, the em interfaces were pushing over 200k interrupts/sec (roughly 60% CPU processing interrupts). So I recompiled the kernel with polling enabled and enabled it on the NICs. The strange thing is that polling shows enabled in ifconfig, but systat -vm still shows the same amount of interrupts. I get the same performance with polling enabled. I'm looking for some guidance on why the machine bogs so much during what seems to me to be something that should barely impact machine performance at all, and also why polling didn't seem to lower the number of interrupts processed. The old machine was 6 years old running an old intel raid5, and it handled NFS and the concurrent file copies without a sweat. My 3ware is setup as follows: a 2 disk mirror, for the system a 4 disk raid10, for /mnt/data1 a 4 disk raid10, for /mnt/data2 Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE-p8 #0: Thu Oct 11 10:43:22 PDT 2007 [EMAIL PROTECTED] :/usr/obj/usr/src/sys/MADONNA Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Genuine Intel(R) CPU @ 2.66GHz (2670.65-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f4 Stepping = 4 Features=0xbfebfbff Features2=0x4e3bd,CX16,,,> AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 2 real memory = 4831838208 (4608 MB) avail memory = 4125257728 (3934 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard lapic0: Forcing LINT1 to edge trigger kbd1 at kbdmux0 ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 acpi_throttle0: on cpu0 acpi_button0: on acpi0 acpi_button1: on acpi0 pcib0: port 0xca2,0xca3,0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 pcib4: irq 17 at device 1.0 on pci2 pci4: on pcib4 pcib5: irq 18 at device 2.0 on pci2 pci5: on pcib5 em0: port 0x3020-0x303f mem 0xf882-0xf883,0xf840-0xf87f irq 18 at device 0.0 on pci5 em0: Ethernet address: 00:15:17:21:bf:30 em1: port 0x3000-0x301f mem 0xf880-0xf881,0xf800-0xf83f irq 19 at device 0.1 on pci5 em1: Ethernet address: 00:15:17:21:bf:31 pcib6: at device 0.3 on pci1 pci6: on pcib6 3ware device driver for 9000 series storage controllers, version: 3.60.02.012 twa0: <3ware 9000 series Storage Controller> port 0x2000-0x203f mem 0xfa00-0xfbff,0xf890-0xf8900fff irq 26 at device 2.0 on pci6 twa0: [GIANT-LOCKED] twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-12, 12 ports, Firmware FE9X 3.08.00.004, BIOS BE9X 3.08.00.002 pcib7: at device 3.0 on pci0 pci7: on pcib7 pcib8: at device 4.0 on pci0 pci8: on pcib8 pcib9: at device 5.0 on pci0 pci9: on pcib9 pcib10: at device 6.0 on pci0 pci10: on pcib10 pcib11: at device 7.0 on pci0 pci11: on pcib11 pci0: at device 8.0 (no driver attached) pcib12: irq 16 at device 28.0 on pci0 pci12: on pcib12 uhci0: port 0x4080-0x409f irq 23 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x4060-0x407f irq 22 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] usb1: on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x4040-0x405f irq 23 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] usb2: on uhci2 usb2: USB revision 1.0 uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered uhci3: port 0x4020-0x403f irq 22 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] usb3: on uhci3 usb3: USB revision 1.0 uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub3: 2 ports with 2 removable, self pow
Interrupt/speed problems with 6.2 NFS server
Hi, I have an new NFS server that is processing roughly 15mbit of NFS traffic that we recently upgraded from an older 4.10 box. It has a 3-ware raid card, and is serving NFS out a single em nic to LAN clients. The machine works great just serving NFS, but when I try to copy data from one raid volume to another for backups, the machine's NFS performance goes way down, and NFSops start taking multiple seconds to perform. The file copy goes quite quickly, as would be expected. The console of the machine also starts to lag pretty badly, and I get the 'typing through mud' effect. I use rdist6 to do the backup. My first impression was that I was having interrupt issues, since during the backup, the em interfaces were pushing over 200k interrupts/sec (roughly 60% CPU processing interrupts). So I recompiled the kernel with polling enabled and enabled it on the NICs. The strange thing is that polling shows enabled in ifconfig, but systat -vm still shows the same amount of interrupts. I get the same performance with polling enabled. I'm looking for some guidance on why the machine bogs so much during what seems to me to be something that should barely impact machine performance at all, and also why polling didn't seem to lower the number of interrupts processed. The old machine was 6 years old running an old intel raid5, and it handled NFS and the concurrent file copies without a sweat. My 3ware is setup as follows: a 2 disk mirror, for the system a 4 disk raid10, for /mnt/data1 a 4 disk raid10, for /mnt/data2 Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE-p8 #0: Thu Oct 11 10:43:22 PDT 2007 [EMAIL PROTECTED] :/usr/obj/usr/src/sys/MADONNA Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Genuine Intel(R) CPU @ 2.66GHz (2670.65-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f4 Stepping = 4 Features=0xbfebfbff Features2=0x4e3bd,CX16,,,> AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 2 real memory = 4831838208 (4608 MB) avail memory = 4125257728 (3934 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard lapic0: Forcing LINT1 to edge trigger kbd1 at kbdmux0 ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 acpi_throttle0: on cpu0 acpi_button0: on acpi0 acpi_button1: on acpi0 pcib0: port 0xca2,0xca3,0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 pcib4: irq 17 at device 1.0 on pci2 pci4: on pcib4 pcib5: irq 18 at device 2.0 on pci2 pci5: on pcib5 em0: port 0x3020-0x303f mem 0xf882-0xf883,0xf840-0xf87f irq 18 at device 0.0 on pci5 em0: Ethernet address: 00:15:17:21:bf:30 em1: port 0x3000-0x301f mem 0xf880-0xf881,0xf800-0xf83f irq 19 at device 0.1 on pci5 em1: Ethernet address: 00:15:17:21:bf:31 pcib6: at device 0.3 on pci1 pci6: on pcib6 3ware device driver for 9000 series storage controllers, version: 3.60.02.012 twa0: <3ware 9000 series Storage Controller> port 0x2000-0x203f mem 0xfa00-0xfbff,0xf890-0xf8900fff irq 26 at device 2.0 on pci6 twa0: [GIANT-LOCKED] twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-12, 12 ports, Firmware FE9X 3.08.00.004, BIOS BE9X 3.08.00.002 pcib7: at device 3.0 on pci0 pci7: on pcib7 pcib8: at device 4.0 on pci0 pci8: on pcib8 pcib9: at device 5.0 on pci0 pci9: on pcib9 pcib10: at device 6.0 on pci0 pci10: on pcib10 pcib11: at device 7.0 on pci0 pci11: on pcib11 pci0: at device 8.0 (no driver attached) pcib12: irq 16 at device 28.0 on pci0 pci12: on pcib12 uhci0: port 0x4080-0x409f irq 23 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x4060-0x407f irq 22 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] usb1: on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x4040-0x405f irq 23 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] usb2: on uhci2 usb2: USB revision 1.0 uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered uhci3: port 0x4020-0x403f irq 22 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] usb3: on uhci3 usb3: USB revision 1.0 uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub3: 2 ports with 2 removable, self pow
amrd disk performance drop after running under high load
Hi. I have 3 Dell 2850 with DELL PERC4 SCSI RAID5 6x300GB running lighttpd serving flash video at around 200Mbit/s. %grep amr /var/run/dmesg.boot amr0: mem 0xf80f-0xf80f,0xfe9c-0xfe9f irq 46 at device 14.0 on pci2 amr0: Using 64-bit DMA amr0: delete logical drives supported by controller amr0: Firmware 521X, BIOS H430, 256MB RAM amr0: delete logical drives supported by controller amrd0: on amr0 amrd0: 1430400MB (2929459200 sectors) RAID 5 (optimal) Trying to mount root from ufs:/dev/amrd0s1a %uname -a FreeBSD ???.ru 6.2-STABLE FreeBSD 6.2-STABLE #2: Mon Oct 8 16:25:20 MSD 2007 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/SMP-amd64-HWPMC amd64 % After some time of running under high load disk performance become expremely poor. At that periods 'systat -vm 1' shows something like this: Disks amrd0 KB/t 85.39 tps 5 MB/s 0.38 % busy 99 It shows 100% load and just 2-10 tps. There's nothing bad in /var/log/messages or 'netstat -m' or 'vmstat -z' or anywhere else. This continues 15 - 30 minutes or so and everything becomes fine again. After some time - 10 - 12 hours it repeats. Apart of all, I tried to make mutex profiling and here's the results (sorted by the total number of acquisitions): Bad case: 102 223514 273977 0 14689 1651568 /usr/src/sys/vm/uma_core.c:2349 (512) 950 263099 273968 0 15004 14427 /usr/src/sys/vm/uma_core.c:2450 (512) 108 150422 175840 0 10978 22988519 /usr/src/sys/vm/uma_core.c:1888 (mbuf) 352 160635 173663 0 10896 9678 /usr/src/sys/vm/uma_core.c:2209 (mbuf) 110 134910 173575 0 10838 9464 /usr/src/sys/vm/uma_core.c:2104 (mbuf) 1104 1335319 106888 12 27 1259 /usr/src/sys/netinet/tcp_output.c:253 (so_snd) 171 77754 97685 0 176 207 /usr/src/sys/net/pfil.c:71 (pfil_head_mtx) 140 77104 97685 0 151 128 /usr/src/sys/netinet/ip_fw2.c:164 (IPFW static rules) 100 76543 97685 0 146 45450 /usr/src/sys/netinet/ip_fw2.c:156 (IPFW static rules) 82 77149 97685 0 243 141221 /usr/src/sys/net/pfil.c:63 (pfil_head_mtx) 1644 914481 97679 9 739 949977 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2320 (ipf filter load/unload mutex) 1642 556643 97679 5 0 0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2455 (ipf filter rwlock) 107 89413 97679 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2142 (ipf cache rwlock) 907 148940 81439 1 3 7447 /usr/src/sys/kern/kern_lock.c:168 (lockbuilder mtxpool) 1764 152282 63435 2 438 336763 /usr/src/sys/net/route.c:197 (rtentry) And in the good case: 1738 821795 553033 1 41 284 /usr/src/sys/netinet/tcp_output.c:253 (so_snd) 2770 983643 490815 2 6 54 /usr/src/sys/kern/kern_lock.c:168 (lockbuilder mtxpool) 106 430941 477500 0 4507 /usr/src/sys/netinet/ip_fw2.c:164 (IPFW static rules) 95 423926 477500 0 4412 5645 /usr/src/sys/netinet/ip_fw2.c:156 (IPFW static rules) 94 427239 477500 0 6323 7453 /usr/src/sys/net/pfil.c:63 (pfil_head_mtx) 82 432359 477500 0 5244 5768 /usr/src/sys/net/pfil.c:71 (pfil_head_mtx) 296 4751550 477498 9 20837 23019 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2320 (ipf filter load/unload mutex) 85 2913118 477498 6 0 0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2455 (ipf filter rwlock) 55 473891 477498 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2142 (ipf cache rwlock) 59 291035 309222 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2169 (ipf cache rwlock) 1627 697811 305094 2 2161 2535 /usr/src/sys/net/route.c:147 (radix node head) 232 804172 305094 2 12193 6500 /usr/src/sys/net/route.c:197 (rtentry) 148 892580 303518 2 594 649 /usr/src/sys/net/route.c:1281 (rtentry) 145 584970 303518 1 13479 11148 /usr/src/sys/net/route.c:1265 (rtentry) 121 282669 303518 0 3529 886 /usr/src/sys/net/if_ethersubr.c:409 (em0) Here you can see that high UMA activity happens in periods of low disk performance. But I'm not sure whether this is a root of the problem, not a consequence. I have similiar servers around doing the same things, and they work fine. Also I had the same problem a year ago with another project and that time nothing helped and i had to install Linux. I can provide additional information regarding this server if needed. What else can I try to solve the problem??? With best regards, Alexey Popov ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
