Re: 8.1 amd64 lockup (maybe zfs or disk related)
On 7 February 2011 20:03, Jeremy Chadwick free...@jdc.parodius.com wrote: They're discussed practically on a monthly basis on the mailing lists (either freebsd-fs or freebsd-stable). Keeping track of them is almost impossible at this point, which is also probably why the Wiki is outdated. I like Sun's take on the matter: http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Tuning_is_Evil Tuning is often evil and should rarely be done. First, consider that the default values are set by the people who know the most about the effects of the tuning on the software that they supply. If a better value exists, it should be the default. While alternative values might help a given workload, it could quite possibly degrade some other aspects of performance. Occasionally, catastrophically so. Which I thing summarise perfectly ZFS tuning. If you want to know 20 differents opinions on how zfs needs to be tuned; talk to 20 different people. Everyone has their own ideas on how it should be done ; believe a particular setting made things better. I tried them all I could read here, none of them make much significant difference, and and when they do, usually it's just for the worse. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Policy on static linking ?
On Friday, 14 January 2011, Pete French petefre...@ticketswitch.com wrote: I build code using static linking for deployment across a set of machines. For me this has a lot of advantages - I know that the code will run, no matter what the state of the ports is on the machine, and if there is a need to upgrade a library then I do it once on the build machine, rebuild the executable, and rsync it out to the leaf nodes. Only one place to track security updates, only one place where I need to have all the porst the code depends on installed. I actually tried to compile a port against another and have it link statically, but I couldn't find a way to do so without hacking the configure script. I was wondering if there was another (and easier) way to do so... I use ldap for authentication purposes, along with pam_ldap and nss_ldap If I compile openldap-client against openssl from ports, then it creates massive problems elsewhere. For example, base ssh server will now crash due to using different libcrypto. compiling ports will also become impossible as bsd tar itself crash (removing ldap call from nsswitch.conf is required to work again) I was then advised in the freebsd forums to uninstall openssl port, compile openldap against openssl base, install it, then re-install openssl port. (I have to use openssl from ports with apache/subversion to fix a bug with TLSv1 making svn commit crash under some circumstances) I dislike this method, because should openldap gets upgraded again and be linked against openssl port, I will lock myself out of the machine again due to sshd crashing. Just like what happened today :( So how can I configure openldap-client to link against libssl and libcrypto statically? Thanks ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Policy on static linking ?
Hi On 15 January 2011 23:48, Jilles Tjoelker jil...@stack.nl wrote: The approach has been used by Debian for some time. Links: http://chris.dzombak.name/blog/2010/03/building-openssl-with-symbol-versioning/ http://chris.dzombak.name/files/openssl/openssl-0.9.8l-symbolVersioning.diff http://rt.openssl.org/Ticket/Display.html?id=1222user=guestpass=guest This sounds very interesting. I do have trouble understanding on how this would make a difference with how it's currently working. base openssl uses libssl.so.6 and libcrypto.so.6 current port openssl is using .so.7 So they too have different sonames; How could changing this to .so.0.9.8 for base and so.1.0.2 for port make things behave differently? JY ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Policy on static linking ?
Hi On 16 January 2011 02:17, Jean-Yves Avenard jyaven...@gmail.com wrote: Hi On 15 January 2011 23:48, Jilles Tjoelker jil...@stack.nl wrote: The approach has been used by Debian for some time. Links: http://chris.dzombak.name/blog/2010/03/building-openssl-with-symbol-versioning/ http://chris.dzombak.name/files/openssl/openssl-0.9.8l-symbolVersioning.diff http://rt.openssl.org/Ticket/Display.html?id=1222user=guestpass=guest This sounds very interesting. I do have trouble understanding on how this would make a difference with how it's currently working. base openssl uses libssl.so.6 and libcrypto.so.6 current port openssl is using .so.7 So they too have different sonames; How could changing this to .so.0.9.8 for base and so.1.0.2 for port make things behave differently? Replying to myself.. Looking at the symbols in the openssl libraries found on a Ubuntu machine, I see what is going on: symbol name: X509_NAME_cmp@OPENSSL_0.9.8 vs X509_NAME_cmp This sounds like a great approach.. and should definitely resolve my problems I think. Are you sure both base and port needs to be patched? I would have assumed that patching only port would be sufficient (provided all tools depending on it are also recompiled) Jean-Yves ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS - moving from a zraid1 to zraid2 pool with 1.5tb disks
Hi On 9 January 2011 19:44, Matthew Seaman m.sea...@infracaninophile.co.uk wrote: Not without backing up your current data, destroying the existing zpool(s) and rebuilding from scratch. Note: raidz2 on 4 disks doesn't really win you anything over 2 x mirror pairs of disks, and the RAID10 mirror is going to be rather more performant. I would have thought that the probability of failure to be slightly different. Sure you out of 4 disks, 2 can fail in both conditions. *But*, in raidz2, any two of the four can fail. In RAID10, the two disks that failed must be in different block otherwise you loose it all As such the resilience for failure in a RAIDz2 is far greater than in a RAID10 system ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS - moving from a zraid1 to zraid2 pool with 1.5tb disks
On 9 January 2011 21:03, Matthew Seaman m.sea...@infracaninophile.co.uk wrote: So you sacrifice performance 100% of the time based on the very unlikely possibility of drives 1+2 or 3+4 failing simultaneously, compared to the similarly unlikely possibility of drives 1+3 or 1+4 or 2+3 or 2+4 But this is not what you first wrote You said the effect were identical. they are not. Now if you want to favour performance over redundancy that's ultimately up to the user... Plus, honestly, the difference in performance between raidz and raid10 is also close to bein insignificant. failing simultaneously?[*] That's not a trade-off worth making IMHO. If the data is that valuable, you should be making copies of it to some independent machine all the time and backing up at frequent intervals, which backups you keep off-site in disaster-proof storage. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
faulted zpool , do not resilver
Hi. I have a raidz2 pool on which I'm trying to replaced two of the drives. it is now showing: [r...@server4 ~]# zpool import pool: pool id: 890764434375195435 state: DEGRADED action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. config: pool DEGRADED raidz2 DEGRADED ada5 ONLINE ada6 ONLINE ada2 ONLINE ada1 ONLINE replacing DEGRADED 1880266799568700877 UNAVAIL cannot open ada4 ONLINE replacing DEGRADED 1406029845814225421 UNAVAIL cannot open ada3 ONLINE Previously, it was resilvering, for 36 hours later it was still not completed and it was showing: Problems with ZFS: pool: pool state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver completed after 307445734561825853h19m with 7 errors on Sat Jan 8 02:19:18 2011 config: NAME STATE READ WRITE CKSUM pool DEGRADED 0 0 7 raidz2 DEGRADED 0 042 ada5 ONLINE 0 0 0 2.74G resilvered ada6 ONLINE 0 0 0 2.60G resilvered ada2 ONLINE 0 0 0 2.74G resilvered ada1 ONLINE 0 0 0 2.60G resilvered replacing DEGRADED 0 0 0 1880266799568700877 UNAVAIL 0 2.57M 0 was /dev/da0 ada4 ONLINE 0 0 0 373G resilvered replacing DEGRADED 0 0 0 1406029845814225421 UNAVAIL 0 2.54M 0 was /dev/da0 ada3 ONLINE 0 0 0 372G resilvered errors: 2 data errors, use '-v' for a list As I obviously didn't want to wait another 307445734561825853h19m (which I believe would be close to the end of the universe). I did a zpool export pool. and did a zpool import once again. Now I can see all the files, everything seems fine.. But no resilvering is occurring and I have waited over an hour now. Usually resilvering would happen after only a few seconds. What should I do? Thank you JY ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: faulted zpool , do not resilver
Hi responding to myself here.. I ran zpool scrub on it. It started to resilver ; but then I got: Problems with ZFS: pool: pool state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver completed after 4h33m with 7 errors on Sun Jan 9 08:23:12 2011 config: NAME STATE READ WRITE CKSUM pool DEGRADED 0 0 9 raidz2 DEGRADED 0 054 ada5 ONLINE 0 0 0 3.07G resilvered ada6 ONLINE 0 0 0 2.92G resilvered ada2 ONLINE 0 0 0 3.07G resilvered ada1 ONLINE 0 0 0 2.92G resilvered replacing DEGRADED 0 0 1 1880266799568700877 UNAVAIL 0 1.92M 0 was /dev/da0 ada4 ONLINE 0 0 0 409G resilvered replacing DEGRADED 0 0 2 1406029845814225421 UNAVAIL 0 1.88M 0 was /dev/da0 ada3 ONLINE 0 0 0 408G resilvered errors: 2 data errors, use '-v' for a list I did zpool export / import ; and I'm back again to: server4# zpool status -v pool: pool state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: none requested config: NAME STATE READ WRITE CKSUM pool DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 ada5 ONLINE 0 0 0 ada6 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada1 ONLINE 0 0 0 replacing DEGRADED 0 0 0 1880266799568700877 UNAVAIL 098 0 was /dev/da0 ada4 ONLINE 0 0 0 replacing DEGRADED 0 0 0 1406029845814225421 UNAVAIL 097 0 was /dev/da0 ada3 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: metadata:0x2013 metadata:0x2014 Any pointers on what I should do now? All my data seems fine :( JY ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: faulted zpool , do not resilver
Hi On 9 January 2011 16:08, Adam Vande More amvandem...@gmail.com wrote: On Sat, Jan 8, 2011 at 10:49 PM, Jean-Yves Avenard jyaven...@gmail.com wrote: Any pointers on what I should do now? All my data seems fine :( The web page mentioned in the error message contains further info, have you reviewed that? (argh, I see it's behind a login now, that's a recent change). This should be it's equivalent: http://dlc.sun.com/osol/docs/content/ZFSADMIN/gbbwl.html I did.. I can't find what this metadata: bit referes to though One way the meta data corruption can occur is the hard drive lies to ZFS when it issues a cache flush. If the device says it did and in reality it didn't and there was a power outage this type of corruption can occur. http://mail.opensolaris.org/pipermail/zfs-discuss/2010-January/035740.html I've been trying for 3 days to resolve those issues now.. I'm going to just destroy the zpool ; I have a full mirror of that zpool on another machine. It's going to be the easiest way. I'm still worried that for the first time I tried replaced disks, it failed so miserably... For some reason moving my SATA disk from on sata interface to another failed. zpool would want to read the pool anymore, complaining that the zpool version was too high (it's version 14, definitely ok). Moving the disks bak to the original sata interface, and I can read the pool just fine. So What I did is unplug the diisk from the old interface, zpool import it (now in degraded mode) and plug back the disk and wait for the resilvering to complete. JY ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: NFSv4 - how to set up at FreeBSD 8.1 ?
On 7 January 2011 08:16, Rick Macklem rmack...@uoguelph.ca wrote: When I said I recalled that they didn't do TCP because of excessive overhead, I forgot to mention that my recollection could be wrong. Also, I suspect you are correct w.r.t. the above statement. (ie. Sun's official position vs something I heard.) Anyhow, appologies if I gave the impression that I was correcting your statement. My intent was just to throw out another statement that I vaguely recalled someone an Sun stating. After hitting yet another serious bug in 8.2 ; I reverted back to 8.1 Interestingly, it now complains about having V4: / in /etc/exports NFSv4 isn't available in 8.1 ? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS - moving from a zraid1 to zraid2 pool with 1.5tb disks
Hi On 7 January 2011 00:45, Daniel Kalchev dan...@digsys.bg wrote: For pure storage, that is a place you send/store files, you don't really need the ZIL. You also need the L2ARC only if you read over and over again the same dataset, which is larger than the available ARC (ZFS cache memory). Both will not be significant for 'backup server' application, because it's very unlikely to do lots of SYNC I/O (where separate ZIL helps), or serve the same files back (where the L2ARC might help). You should also know that having large L2ARC requires that you also have larger ARC, because there are data pointers in the ARC that point to the L2ARC data. Someone will do good to the community to publish some reasonable estimates of the memory needs, so that people do not end up with large but unusable L2ARC setups. It seems that the upcoming v28 ZFS will help greatly with the ZIL in the main pool.. yes, it made a *huge* difference for me.. It went from way too slow to comprehend what's going on to still slow but I can live with it and I found no significant difference between ZIL on the main pool and on a separate SSD ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS - moving from a zraid1 to zraid2 pool with 1.5tb disks
On 6 January 2011 22:26, Chris Forgeron cforge...@acsi.ca wrote: You know, these days I'm not as happy with SSD's for ZIL. I may blog about some of the speed results I've been getting over the last 6mo-1yr that I've been running them with ZFS. I think people should be using hardware RAM drives. You can get old Gigabyte i-RAM drives with 4 gig of memory for the cost of a 60 gig SSD, and it will trounce the SSD for speed. I'd put your SSD to L2ARC (cache). Where do you find those though. I've looked and looked and all references I could find was that battery-powered RAM card that Sun used in their test setup, but it's not publicly available.. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS - moving from a zraid1 to zraid2 pool with 1.5tb disks
On 7 January 2011 12:42, Jeremy Chadwick free...@jdc.parodius.com wrote: DDRdrive: http://www.ddrdrive.com/ http://www.engadget.com/2009/05/05/ddrdrives-ram-based-ssd-is-snappy-costly/ ACard ANS-9010: http://techreport.com/articles.x/16255 GC-RAMDISK (i-RAM) products: http://us.test.giga-byte.com/Products/Storage/Default.aspx Be aware these products are absurdly expensive for what they offer (the cost isn't justified), not to mention in some cases a bottleneck is imposed by use of a SATA-150 interface. I'm also not sure if all of them offer BBU capability. Why not one of those SSD PCIe card that gives over 500MB/s read and write. And they aren't too expensive either... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: NFSv4 - how to set up at FreeBSD 8.1 ?
Hi On 4 January 2011 10:50, Rick Macklem rmack...@uoguelph.ca wrote: If the above 2 lines are in your /etc/exports file and / is a ufs file system, then the above should work. For a zfs / you must either: - export / as well as /data or - use v4: /data so that the nfsv4 root is at /data Also, make sure you are running the experimental server: - either start both mountd and nfsd with the -e option or specify nfsv4_server_enable=YES nfs_server_enable=YES in your /etc/rc.conf. Also, you need to create an empty /var/db/nfs-stablerestart file before the experimental NFS server will start up the first time. (A fix for that is in the works, but isn't even in head yet.) Try looking at man nfsv4 and checking that the daemons are running and that nothing got logged in /var/log/messages when they started up. Good luck with it, rick After reading this thread, I tried NFSv4 today.. Whenever I tried to mount from a linux client, I get: mount -o vers=4 server4:/pool/backup/sites/m /mnt NFS compound failed for server server4: error 7 (RPC: Authentication error) NFS compound failed for server server4: error 7 (RPC: Authentication error) NFS compound failed for server server4: error 7 (RPC: Authentication error) NFS compound failed for server server4: error 7 (RPC: Authentication error) NFS compound failed for server server4: error 7 (RPC: Authentication error) NFS compound failed for server server4: error 7 (RPC: Authentication error) nfs mount: mount: /mnt: Permission denied with NFS v3 it mounts just fine any ideas? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: NFSv4 - how to set up at FreeBSD 8.1 ?
Hi On 5 January 2011 12:09, Rick Macklem rmack...@uoguelph.ca wrote: You can also do the following: For /etc/exports V4: / /usr/home -maproot=root -network 192.168.183.0 -mask 255.255.255.0 Then mount: # mount_nfs -o nfsv4 192.168.183.131:/usr/home /marek_nfs4/ (But only if the file system for / is ufs and not zfs and, admittedly there was a debate that has to be continued someday that might make it necessary to export / as well for ufs like zfs requires.) rick ps: And some NFSv4 clients can cross server mount points, unlike NFSv2, 3. I've done that (exporting V4: /) but then when I mount a sub zfs filesystem (e.g. /pool/backup/sites/m) then it appears empty on the client. If I export /pool/backup/sites/m , then I see the content of the directory. Most of the sub-directory in /pool are actually zfs file system mounted. It is something I expected with NFSv3 .. but not with nfs v4. JY ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS - moving from a zraid1 to zraid2 pool with 1.5tb disks
On 2 January 2011 02:11, Damien Fleuriot m...@my.gd wrote: I remember getting rather average performance on v14 but Jean-Yves reported good performance boosts from upgrading to v15. that was v28 :) saw no major difference between v14 and v15. JY ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE: Kernel Panic
On Wednesday, 29 December 2010, jhell jh...@dataix.net wrote: Another note too, I think I read that you mentioned using the L2ARC and slog device on the same disk You simply shouldn't do this it could be contributing to the real cause and there is absolutely no gain in either sanity or performance and you will end up bottle-necking your system. And why would that be? I've read so many conflictinginformation on the matter over the past few days that I'm starting to wonder if there's an actual definitive answer on the matter or if anyone has a clue regarding what they're talking about. It ranges from, should only use raw disk to freebsd isn't solaris so slices are fine. Don't use slice because they can't be read by another OS use partitions.. It doesn't apply to SSD and so on.. The way I look at it, the only thing that would bottleneck access to that SSD drive, is the SATA interface itself. So using two drives, or two partitions on the same drive, I can't see how it would make much difference if any other than the traditional I think I know argument. Surely latency as with know it with hard drive do not apply to SSDs. Even within sun's official documentation, they are contradicting information, starting from the commands on how to add remove/cache of log device. It seems to me that tuning ZFS is very much like black magic, everyone has their own idea about what to do, and not once did I get to read conclusive evidence about what is best or find an information people actually agree on. As for using unofficial code, sure I accept that risk now. I made a conscious decision on using it, there's now no way to go back and I accept that. At the end of the day, it's the only thing that will make that code suitable for real world condition: testing. If that particular code isn't put under any actual stress how else are you going to know if its good or not. I don't really like reading between the lines of your post that I shouldn't be surprised should anything break or that it doesn't matter if it crashes. there's a deadlock occurring somewhere : it needs to be found. I know nothing about the ZFS code, and I could only do what I'm capable of under those circumstances: find a way to reproduce the problem consistently, report as much information as I have so someone more clueey will know what to do with it. Hope that makes sense Jean-Yves ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: slow ZFS on FreeBSD 8.1
On Thursday, 30 December 2010, Matthew Seaman m.sea...@infracaninophile.co.uk wrote: No -- the on-disk format is different. ZFS will run fine with the older on-disk formats, but you won't get the full benefits without updating them. You'll need to run both 'zpool update -a' and 'zfs update -a' -- I believe it's upgrade you want to use, not update. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE: ARRRGG HELP !!
Well Today I added the log device: zpool add pool log /dev/ada1s1 (8GB slice on a SSD Intel X25 disk).. then added the cache (32GB) zpool add pool cache /dev/ada1s2 So far so good. zpool status - all good. Reboot : it hangs booted in single user mode, zpool status: ZFS filesystem version 5 ZFS storage pool version 28 and that's it no more.. Just like before when I thought that removing the log disk had failed. This time no error nothing... just a nasty hang and unusable system again... :( ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE: Kernel Panic
Hi On 27 December 2010 16:04, jhell jh...@dataix.net wrote: 1) Set vfs.zfs.recover=1 at the loader prompt (OK set vfs.zfs.recover=1) 2) Boot into single user mode without opensolaris.ko and zfs.ko loaded 3) ( mount -w / ) to make sure you can remove and also write new zpool.cache as needed. 3) Remove /boot/zfs/zpool.cache 4) kldload both zfs and opensolaris i.e. ( kldload zfs ) should do the trick 5) verify that vfs.zfs.recover=1 is set then ( zpool import pool ) 6) Give it a little bit monitor activity using Ctrl+T to see activity. Ok.. I've got into the same situation again, no idea why this time. I've followed your instructions, and sure enough I could do an import of my pool again. However, wanted to find out what was going on.. So I did: zpool export pool followed by zpool import And guess what ... hanged zpool again.. can't Ctrl-C it, have to reboot.. So here we go again. Rebooted as above. zpool import pool - ok this time, I decided that maybe that what was screwing things up was the cache. zpool remove pool ada1s2 - ok zpool status: # zpool status pool: pool state: ONLINE scan: scrub repaired 0 in 18h20m with 0 errors on Tue Dec 28 10:28:05 2010 config: NAMESTATE READ WRITE CKSUM poolONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada2ONLINE 0 0 0 ada3ONLINE 0 0 0 ada4ONLINE 0 0 0 ada5ONLINE 0 0 0 ada6ONLINE 0 0 0 ada7ONLINE 0 0 0 logs ada1s1ONLINE 0 0 0 errors: No known data errors # zpool export pool - ok # zpool import pool - ok # zpool add pool cache /dev/ada1s2 - ok # zpool status pool: pool state: ONLINE scan: scrub repaired 0 in 18h20m with 0 errors on Tue Dec 28 10:28:05 2010 config: NAMESTATE READ WRITE CKSUM poolONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada2ONLINE 0 0 0 ada3ONLINE 0 0 0 ada4ONLINE 0 0 0 ada5ONLINE 0 0 0 ada6ONLINE 0 0 0 ada7ONLINE 0 0 0 logs ada1s1ONLINE 0 0 0 cache ada1s2ONLINE 0 0 0 errors: No known data errors # zpool export pool - ok # zpool import load: 0.00 cmd: zpool 405 [spa_namespace_lock] 15.11r 0.00u 0.03s 0% 2556k load: 0.00 cmd: zpool 405 [spa_namespace_lock] 15.94r 0.00u 0.03s 0% 2556k load: 0.00 cmd: zpool 405 [spa_namespace_lock] 16.57r 0.00u 0.03s 0% 2556k load: 0.00 cmd: zpool 405 [spa_namespace_lock] 16.95r 0.00u 0.03s 0% 2556k load: 0.00 cmd: zpool 405 [spa_namespace_lock] 32.19r 0.00u 0.03s 0% 2556k load: 0.00 cmd: zpool 405 [spa_namespace_lock] 32.72r 0.00u 0.03s 0% 2556k load: 0.00 cmd: zpool 405 [spa_namespace_lock] 40.13r 0.00u 0.03s 0% 2556k ah ah ! it's not the separate log that make zpool crash, it's the cache ! Having the cache in prevent from importing the pool again rebooting: same deal... can't access the pool any longer ! Hopefully this is enough hint for someone to track done the bug ... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE: ARRRGG HELP !!
On 28 December 2010 08:56, Freddie Cash fjwc...@gmail.com wrote: Is that a typo, or the actual command you used? You have an extra s in there. Should be log and not logs. However, I don't think that command is correct either. I believe you want to use the detach command, not remove. # zpool detach pool label/zil well, I tried the detach command: server4# zpool detach pool ada1s1 cannot detach ada1s1: only applicable to mirror and replacing vdevs server4# zpool remove pool ada1s1 server4# so you need to use remove, and adding log (or cache) makes no difference whatsoever.. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE: Kernel Panic
On 29 December 2010 03:15, Jean-Yves Avenard jyaven...@gmail.com wrote: # zpool import load: 0.00 cmd: zpool 405 [spa_namespace_lock] 15.11r 0.00u 0.03s 0% 2556k load: 0.00 cmd: zpool 405 [spa_namespace_lock] 15.94r 0.00u 0.03s 0% 2556k load: 0.00 cmd: zpool 405 [spa_namespace_lock] 16.57r 0.00u 0.03s 0% 2556k load: 0.00 cmd: zpool 405 [spa_namespace_lock] 16.95r 0.00u 0.03s 0% 2556k load: 0.00 cmd: zpool 405 [spa_namespace_lock] 32.19r 0.00u 0.03s 0% 2556k load: 0.00 cmd: zpool 405 [spa_namespace_lock] 32.72r 0.00u 0.03s 0% 2556k load: 0.00 cmd: zpool 405 [spa_namespace_lock] 40.13r 0.00u 0.03s 0% 2556k ah ah ! it's not the separate log that make zpool crash, it's the cache ! Having the cache in prevent from importing the pool again rebooting: same deal... can't access the pool any longer ! Hopefully this is enough hint for someone to track done the bug ... More details as I was crazy enough to try various things. The problem of zpool being stuck in spa_namespace_lock, only occurs if you are using both the cache and the log at the same time. Use one or the other : then there's no issue But the instant you add both log and cache to the pool, it becomes unusable. Now, I haven't tried using cache and log from a different disk. The motherboard on the server has 8 SATA ports, and I have no free port to add another disk. So my only option to have both a log and cache device in my zfs pool, is to use two slices on the same disk. Hope this helps.. Jean-Yves ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE: Kernel Panic
Hi On Wednesday, 29 December 2010, Martin Matuska m...@freebsd.org wrote: Please don't consider these patches as production-ready. What we want to do is find and resolve as many bugs as possible. To help us fix these bugs, a way to reproduce the bug from a clean start (e.g. in virtualbox) would be great and speed up finding the cause for the problem. Your problem looks like some sort of deadlock. In your case, when you experiene the hang, try running procstat -k -k PID in another shell (console). That will give us valuable information. I am away until next week now (hopefully no problem will occur until then) I will try reproduce the issue then. I have to say that v28 massively increased write performance over samba, over 3 times than when v14 or v15. How do you disable ZIL with v28? I wanted to test performance in the case I'm trying to qtroubleshoot. Writing our file over samba, V14-v15: 55s V14-v15 zip disabled: 6s V28: 16s.. (with or without separate log drive: SSD Intel X25-m 40GB). Playing with the only zil parameter showing in sysctl, made no difference whatsoever. UFS boot drive: 14s sequential read shows over 280MB/s from that raidz array, similar with writes I started a thread in the freebsd forum: http://forums.freebsd.org/showthread.php?t=20476 And finally, which patch should I try on your site? Thanks ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE: ARRRGG HELP !!
Hi On Tuesday, 28 December 2010, Freddie Cash fjwc...@gmail.com wrote: On Sun, Dec 26, 2010 at 4:43 PM, Jean-Yves Avenard jyaven...@gmail.com wrote: On 27 December 2010 09:55, Jean-Yves Avenard jyaven...@gmail.com wrote: Hi there. I used stable-8-zfsv28-20101223-nopython.patch.xz from http://people.freebsd.org/~mm/patches/zfs/v28/ I did the following: # zpool status pool: pool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM pool ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada4 ONLINE 0 0 0 ada5 ONLINE 0 0 0 ada6 ONLINE 0 0 0 ada7 ONLINE 0 0 0 cache label/zcache ONLINE 0 0 0 errors: No known data errors so far so good [r...@server4 /pool/home/jeanyves_avenard]# zpool add pool log /dev/label/zil [r...@server4 /pool/home/jeanyves_avenard]# zpool status pool: pool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM pool ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada4 ONLINE 0 0 0 ada5 ONLINE 0 0 0 ada6 ONLINE 0 0 0 ada7 ONLINE 0 0 0 logs label/zil ONLINE 0 0 0 cache label/zcache ONLINE 0 0 0 errors: No known data errors so far so good: # zpool remove pool logs label/zil cannot remove logs: no such device in pool Is that a typo, or the actual command you used? You have an extra s in there. Should be log and not logs. However, I don't think that command is correct either. I believe you want to use the detach command, not remove. # zpool detach pool label/zil -- Freddie Cash fjwc...@gmail.com It was a typo, it should have been log (according to sun's doc). As it was showing logs in the status I typed this. According to sun, it zpool remove pool cache/log A typo should have never resulted in what happened, showing an error for sure; but zpool hanging and kernel panic? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE
Hi there. I used stable-8-zfsv28-20101223-nopython.patch.xz from http://people.freebsd.org/~mm/patches/zfs/v28/ simply because it was the most recent at this location. Is this the one to use? Just asking cause the file server I installed it on has stopped responding this morning and doing a remote power cycle didn't work. So got to get to the office and see what went on :( Suspect a kernel panic of some kind Jean-Yves ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE: ARRRGG HELP !!
On 27 December 2010 09:55, Jean-Yves Avenard jyaven...@gmail.com wrote: Hi there. I used stable-8-zfsv28-20101223-nopython.patch.xz from http://people.freebsd.org/~mm/patches/zfs/v28/ I did the following: # zpool status pool: pool state: ONLINE scan: none requested config: NAMESTATE READ WRITE CKSUM poolONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada2ONLINE 0 0 0 ada3ONLINE 0 0 0 ada4ONLINE 0 0 0 ada5ONLINE 0 0 0 ada6ONLINE 0 0 0 ada7ONLINE 0 0 0 cache label/zcache ONLINE 0 0 0 errors: No known data errors so far so good [r...@server4 /pool/home/jeanyves_avenard]# zpool add pool log /dev/label/zil [r...@server4 /pool/home/jeanyves_avenard]# zpool status pool: pool state: ONLINE scan: none requested config: NAMESTATE READ WRITE CKSUM poolONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada2ONLINE 0 0 0 ada3ONLINE 0 0 0 ada4ONLINE 0 0 0 ada5ONLINE 0 0 0 ada6ONLINE 0 0 0 ada7ONLINE 0 0 0 logs label/zil ONLINE 0 0 0 cache label/zcache ONLINE 0 0 0 errors: No known data errors so far so good: # zpool remove pool logs label/zil cannot remove logs: no such device in pool ^C Great... now nothing respond.. Rebooting the box, I can boot in single user mode. but doing zpool status give me: ZFS filesystem version 5 ZFS storage pool version 28 and it hangs there forever... What should I do :( ? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE: ARRRGG HELP !!
Rebooting in single-user mode. zpool status pool or spool scrub pool hangs just the same ... and there's no disk activity either ... Will download a liveCD of OpenIndiana, hopefully it will show me what's wrong :( Jean-Yves ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE: Kernel Panic
tried to force a zpool import got a kernel panic: panic: solaris assert: weight = space weight = 2 * space, file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c, line: 793 cpuid = 5 KDB: stack backtrace #0: 0xff805f64be at kdb_backtrace #1 .. panic+0x187 #2 .. metaslab_weight+0xe1 #3: metaslab_sync_done+0x21e #4: vdev_sync_done #5: spa_sync+0x6a2 #6 txg_sync_thread+0x147 #7: fork_exit+0x118 #8: fork_trampoline+0xe uptime 2m25s.. sorry for not writing down all the RAM addressed in the backtrace ... Starting to smell very poorly :( ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE: Kernel Panic
Responding to myself again :P On 27 December 2010 13:28, Jean-Yves Avenard jyaven...@gmail.com wrote: tried to force a zpool import got a kernel panic: panic: solaris assert: weight = space weight = 2 * space, file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c, line: 793 cpuid = 5 KDB: stack backtrace #0: 0xff805f64be at kdb_backtrace #1 .. panic+0x187 #2 .. metaslab_weight+0xe1 #3: metaslab_sync_done+0x21e #4: vdev_sync_done #5: spa_sync+0x6a2 #6 txg_sync_thread+0x147 #7: fork_exit+0x118 #8: fork_trampoline+0xe uptime 2m25s.. Command used to import in FreeBSD was: zpool import -fF -R / pool which told me that zil was missing, and to use -m I booted openindiana (which is the only distribution I could ifnd with a live CD supporting zpool v28) Doing a zpool import actually made it show that the pool had successfully been repaired by the command above. It did think that the pool was in use (and it was, as I didn't do a zpool export). So I run zpool import -f pool in openindiana, and luckily, all my files were there. Not sure if anything was lost... in openindiana, I then ran zpool export and rebooted into FreeBSD. I ran zpool import there, and got the same original behaviour of a zpool import hanging, I can't sigbreak it nothing. Only left with the option of rebooting. Back into openindiana, tried to remove the log drive, but no luck. Always end up with the message: cannot remove log: no such device in pool Googling that error seems to be a common issue when trying to remove a ZIL but while that message is displayed, the log drive is actually removed. Not in my case.. So I tried something brave: In Open Indiana zpool export pool rebooted the PC, disconnected the SSD drive I had use and rebooted into openindiana ran zpool import -fF -R / pool (complained that log device was missing) and again zpool import -fF -m -R / pool zfs status showed that logs device being unavailable this time. ran zpool remove pool log hex_number_showing_in_place It showed the error cannot remove log: no such device in pool but zpool status showed that everything was allright zpool export pool , then reboot into FreeBSD zpool import this time didn't hang and successfully imported my pool. All data seems to be there. Summary: v28 is still buggy when it comes to removing the log device... And once something is screwed, zpool utility becomes hopeless as it hangs. So better have a OpenIndiana live CD to repair things :( But I won't be trying to remove the log device for a long time ! at least the data can be recovered when it happens.. Could it be that this is related to the v28 patch I used (http://people.freebsd.org/~mm/patches/zfs/v28/stable-8-zfsv28-20101223-nopython.patch.xz and should have stuck to the standard one). Jean-Yves Breezing again ! ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: New ZFSv28 patchset for 8-STABLE: Kernel Panic
Hi On 27 December 2010 16:04, jhell jh...@dataix.net wrote: Before anything else can you: (in FreeBSD) 1) Set vfs.zfs.recover=1 at the loader prompt (OK set vfs.zfs.recover=1) 2) Boot into single user mode without opensolaris.ko and zfs.ko loaded 3) ( mount -w / ) to make sure you can remove and also write new zpool.cache as needed. 3) Remove /boot/zfs/zpool.cache 4) kldload both zfs and opensolaris i.e. ( kldload zfs ) should do the trick 5) verify that vfs.zfs.recover=1 is set then ( zpool import pool ) 6) Give it a little bit monitor activity using Ctrl+T to see activity. You should have your pool back to a working condition after this. The reason why oi_127 can't work with your pool is because it cannot see FreeBSD generic labels. The only way to work around this for oi_127 would be to either point it directly at the replacing device or to use actual slices or partitions for your slogs and other such devices. Use adaNsN or gpt or gptid for working with your pool if you plan on using other OS's for recovery effects. Hi.. Thank you for your response, I will keep it safely should it ever occur again. Let me explain why I used labels.. It all started when I was trying to solve some serious performance issue when running with zfs http://forums.freebsd.org/showthread.php?t=20476 One of the step in trying to trouble shoot the latency problem, was to use AHCI ; I had always thought that activating AHCI in the bios was sufficient to get it going on FreeBSD, turned out that was the case and that I needed to load ahci.ko as well. After doing so, my system wouldn't boot anymore as it was trying to be /dev/ad0 which didn't exist anymore and was now names /dev/ata0. So I used a label to the boot disk to ensure that I will never encounter that problem ever again. In the same mindset, I used labels for the cache and log device I later added to the pool... I have to say however, that zfs had no issue using the labels until I tried to remove it. I had rebooted several times without having any problems. zpool status never hanged It all started to play up when I ran the command: zpool remove pool log label/zil zpool never ever came out from running that command (I let it run for a good 30 minutes, during which I was fearing the worse, and once I rebooted and nothing ever worked, suicide looked like an appealing alternative) It is very disappointing however that because the pool is in a non-working state, none of the command available to troubleshoot the problem would actually work (which I'm guessing is related to zpool looking for a device name that it can never find being a label) I also can't explain why FreeBSD would kernel panic when it was finally in a state of being able to do an import. I have to say unfortunately, that if I hadn't had OpenIndiana, I would probably still by crying underneath my desk right now... Thanks again for your email, I have no doubt that this would have worked but in my situation, I got your answer in just 2 hours, which is better than any paid support could provide ! Jean-Yves PS: saving my 5MB files over the network , went from 40-55s with v15 to a constant 16s with v28... I can't test with ZIL completely disabled , it seems that vfs.zfs.zil_disable has been removed, and so did vfs.zfs.write_limit_override ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: twa breakage on AMD64 with9.1.5.23wareversionand2005-01-1103:00:49 UTC RELENG_5 commit
can't wait to test.. I'm just about to revert to FreeBSD i386 this week-end as i can't put up with the constant amd64 crashes anymore... It's not just the twa driver ; the NFS server often start to take 100% of CPU time and various network utilities crashes once a week. I didn't notice much speed increase when I moved from i386 to amd64 so it's not going to be too much of an issue to revert back (except Subversion server: for some reasons it's much faster on amd64) Jean-Yves On 04/03/2005, at 7:50 AM, Scott Long wrote: Jean-Yves Avenard wrote: Well, it may work for you, but not for me. Still hangs exactly in the same spot. Jean-Yves On 21/01/2005, at 12:24 AM, Michael Meltzer wrote: looks like patch3 worked, good job! I had to turn off TWA_DEBUG to get the boot going, the debug was starving the boot on a 9600 baud terminal. I can increase the speed and caputer the output on monday if it will help(it on the large side, bigger than the 10,000 line capture buffer in am using). I incude 2 iozone reports, the first one the new driver the second one the old driver, Is thier a way to increase the read ahead in freebsd?? -mjm --- I have a patch that I'll be posting for review tonight that hopefully will put all of this to rest. Stay tuned, and harass me if you don't see it by tomorrow. Scott --- Jean-Yves Avenard Hydrix Pty Ltd - Embedding the net www.hydrix.com | fax +61 3 9572 2686 | phone +61 3 9572 0686 ext 100 VoIP: direct: [EMAIL PROTECTED], general: [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: twa breakage on AMD64 with9.1.5.23wareversionand2005-01-1103:00:49 UTC RELENG_5 commit
Well, it may work for you, but not for me. Still hangs exactly in the same spot. Jean-Yves On 21/01/2005, at 12:24 AM, Michael Meltzer wrote: looks like patch3 worked, good job! I had to turn off TWA_DEBUG to get the boot going, the debug was starving the boot on a 9600 baud terminal. I can increase the speed and caputer the output on monday if it will help(it on the large side, bigger than the 10,000 line capture buffer in am using). I incude 2 iozone reports, the first one the new driver the second one the old driver, Is thier a way to increase the read ahead in freebsd?? -mjm --- Jean-Yves Avenard Hydrix Pty Ltd - Embedding the net www.hydrix.com | fax +61 3 9572 2686 | phone +61 3 9572 0686 ext 100 VoIP: direct: [EMAIL PROTECTED], general: [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: twa breakage on AMD64 with 9.1.5.2 3wareversionand2005-01-1103:00:49 UTC RELENG_5 commit
Hello On 15/01/2005, at 12:40 PM, Vinod Kashyap wrote: Could you give the attached patch a try? Just a hunch... I did... same behavior. it just hangs right after displaying the twa0 line. Actually, hangs is incorrect. As I had to drive to the office to reload a previous kernel (never, ever rely on nextboot my friends!) When I got there I saw that it went a little bit further, so it seems that it doesn't just hang... it is just very very slow Jean-Yves --- Jean-Yves Avenard Hydrix Pty Ltd - Embedding the net www.hydrix.com | fax +61 3 9572 2686 | phone +61 3 9572 0686 ext 100 VoIP: direct: [EMAIL PROTECTED], general: [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: twa breakage on AMD64 with 9.1.5.2 3wareversionand2005-01-1103:00:49 UTC RELENG_5 commit
From there it just hangs... The drive activity LED will be lit for about 10s and after that goes off. The more it goes, the more I'm thinking of reverting to i386 mode, the amd64 just keeps crashing about once a fortnight. Jean-Yves --- Jean-Yves Avenard Hydrix Pty Ltd - Embedding the net www.hydrix.com | fax +61 3 9572 2686 | phone +61 3 9572 0686 ext 100 VoIP: direct: [EMAIL PROTECTED], general: [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: twa breakage on AMD64 with 9.1.5.2 3wareversionand2005-01-1103:00:49 UTC RELENG_5 commit
On 16/01/2005, at 4:21 AM, Vinod Kashyap wrote: You seem to running twa version 2.50.00.000. This is the driver prior to the recent update. Do you see the hang even with this? The latest version is 2.50.02.011. What do you mean even with this? Oops I see ; sorry. I just wanted to point where the hang was, the dump I posted was made with a Kernel compiled back in November (still from RELENG_5). Obviously I can't give you a dump of the new kernel has when booting the kernel compiler last Friday I lost the access to it I haven't had the chance to try your patch yet; i don't have physical access to the server until Monday and i don't want to take the risk of starting a kernel that will hang. --- Jean-Yves Avenard Hydrix Pty Ltd - Embedding the net www.hydrix.com | fax +61 3 9572 2686 | phone +61 3 9572 0686 ext 100 VoIP: direct: [EMAIL PROTECTED], general: [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
