Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
Just to let everyone know, Greg's going to be out-of-town for a little while and won't be able to continue debugging. I'm taking a crack at it again. So far, it seems to be a combination of softupdates+vinum+raid5 that triggers the bug, whatever it is. If I turn off softupdates the raid5 volume seems happy, with and without async. It seems likely to be some odd interaction between the raid5 code and softupdates, since the simpler mirror/striping seems to work fine with it. Ring any bells for anyone? -- Parag Patel To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Help needed with debugging (was: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates)
On Monday, 30 August 1999 at 7:53:12 +0200, Bernd Walter wrote: On Sun, Aug 29, 1999 at 04:48:32PM -0700, Matthew Dillon wrote: : :How similar? The trap above is extremely bad; it looks like a return :on a corrupted stack or a jump through a null function vector. : :Make very sure that your vinum kld is in sync with your kernel. This looks like an indirect call through a NULL function pointer. In my case it is a call to bp-b_iodone in kern/vfs_bio.c:2580 which is 0 :( We seem to have a specific problem here. To summarize: 1. It was first reported (by Bernd) on 15 August. 2. The system crashes in biodone because the buffer header has the B_CALL flag set, but the value of bp-b_iodone is NULL. 3. bp-b_iodone is only set to NULL (well, 0) in one place: getnewbuf, which I don't call. It gets reset to a previous value in dsiodone, which is about the only place where things could conceivably go wrong. I put some check code in at every conceivable place, and the only place which found the situation was in biodone. 4. In every case, the fields just before b_iodone were also zeroed: b_dev = 0xc098d840, b_data = 0xc0befc00 "\2275\t", b_kvabase = 0x0, b_kvasize = 0x0, b_lblkno = 0x0, b_blkno = 0x0, b_offset = 0x0, b_iodone = 0, b_iodone_chain = 0x0, b_vp = 0xc5d4a700, The b_dev and b_vp fields are OK, and b_data looks OK as well. I'm guessing that something is overwriting some of the fields in the header, but it's always the same, and I can't find anything in the code that does that. 5. In one case yesterday, there were two requests involved in the vinum request. The other one had already completed (been through biodone), but the bp-b_iodone word was zeroed out in the same manner. In addition, other fields have been set by Vinum's iodone function. From this I deduce that the fields were zeroed after biodone, which makes it very unlikely that it was done by Vinum. I don't know how to proceed at the moment. Matt Dillon has suggested adding some dummy fields in the buffer header and setting them to known values, but I expect this will drive the problem into hiding. Instead, I'm migrating the whole thing to -STABLE to see if it happens there. In view of the impending release of 3.3, this makes sense anyway. In the meantime, if anybody has any ideas, or if any of this rings a bell, I'd be grateful for feedback. Greg -- See complete headers for address, home page and phone numbers finger [EMAIL PROTECTED] for PGP public key To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
On Mon, Aug 30, 1999 at 09:14:11AM +0200, Bernd Walter wrote: On Mon, Aug 30, 1999 at 03:58:16PM +0930, Greg Lehey wrote: Yes, this is the same thing. Until Parag came along, I was beginning to think it was a problem with your hardware :-( Yes - I would have done tests with a host at work to check if it's reproduceable with another hardware - now I think this is obsolete. It would be difficult to give you login to that host and a host for gdb anyway. The host at work paniced too. It's a dual CPU PIII system - but I hadn't DDB and everthing else needed compiled in. Is there any way for me to look at this? Do you have an IDE disk on your machine that you could dump to? Will it write a dump to ide? I have only one ide cd-rom in that host, but I have an unused 130M IDE-HDD. OK I set up the 130M drive configured it as dumpdev and used your patches. One patch did not compile - I wrote you in a different mail. After some time of waiting I got a panic and tried to dump - but it did not work. dumping hang after syncing the drives - is there a way to directly initiate the dump without syncing? I will reproduce the panic and leave it there, so that you can check with kgdb. -- B.Walter COSMO-Project http://www.cosmo-project.de [EMAIL PROTECTED] Usergroup[EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
On Sun, Aug 29, 1999 at 12:43:13PM -0700, Parag Patel wrote: -ddb crash output- Fatal trap 12: page fault while in kernel mode mp_lock = 0303; cpuid = 3; lapic.id = 0200 fault virtual address = 0x0 fault code = supervisor read, page not present instruction pointer = 0x8:0x0 stack pointer = 0x10:0xd5730b08 frame pointer = 0x10:0xd5730b4c code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 376 (cpio) interrupt mask = bio - SMP: XXX kernel: type 12 trap, code=0 Stopped at 0: [...] This looks similar to the panics I got since some days. I was not able to dump too, but was successfull with remote kgdb My system is single CPU with vinum and softupdates. I am able to reproduce the panic during several minutes. Maybe Greg will take a look at it - I cc'ed to him. -- B.Walter COSMO-Project http://www.cosmo-project.de [EMAIL PROTECTED] Usergroup[EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
On Sun, Aug 29, 1999 at 12:43:13PM -0700, Parag Patel wrote: -ddb crash output- Fatal trap 12: page fault while in kernel mode mp_lock = 0303; cpuid = 3; lapic.id = 0200 fault virtual address = 0x0 fault code = supervisor read, page not present instruction pointer = 0x8:0x0 stack pointer = 0x10:0xd5730b08 frame pointer = 0x10:0xd5730b4c code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 376 (cpio) interrupt mask = bio - SMP: XXX kernel: type 12 trap, code=0 Stopped at 0: [...] This looks similar to the panics I got since some days. How similar? The trap above is extremely bad; it looks like a return on a corrupted stack or a jump through a null function vector. Make very sure that your vinum kld is in sync with your kernel. -- \\ The mind's the standard \\ Mike Smith \\ of the man. \\ [EMAIL PROTECTED] \\-- Joseph Merrick \\ [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
On Sunday, 29 August 1999 at 18:08:33 -0700, Parag Patel wrote: On Sun, 29 Aug 1999 16:48:32 PDT, Matthew Dillon wrote: This looks like an indirect call through a NULL function pointer. Wow - I'm impressed! You really know your kernel debugging! :) It is a null-pointer dereference that's crashing it. Here's the gdb stack back-trace: (gdb) bt #0 0x0 in ?? () #1 0xc017afc3 in biodone (bp=0xc1550c18) at ../../kern/vfs_bio.c:2580 #2 0xc0126c2a in dadone (periph=0xc13de980, done_ccb=0xc163a000) at ../../cam/scsi/scsi_da.c:1295 #3 0xc01228e7 in camisr (queue=0xc02bd854) at ../../cam/cam_xpt.c:6223 #4 0xc01226f9 in swi_cambio () at ../../cam/cam_xpt.c:6130 The contents of the "bp" pointer are dumped below. Sure enough, bp-b_iodone is indeed NULL, but I have no idea how or why. Perhaps it is the flags that are incorrectly stating it is a B_CALL? I can print out other structs and such - I'll leave the system as-is for now. Thanks! The stack looks intact ... look at the sp verses the frame pointer. If the 'trace' command is resulting in a panic, perhaps it is because there is no new stack frame. Giving the trace command an argument will help. (gdb) up #1 0xc017afc3 in biodone (bp=0xc1550c18) at ../../kern/vfs_bio.c:2580 2580(*bp-b_iodone) (bp); (gdb) l 2575} 2576 2577/* call optional completion function if requested */ 2578if (bp-b_flags B_CALL) { 2579bp-b_flags = ~B_CALL; 2580(*bp-b_iodone) (bp); 2581splx(s); 2582return; 2583} 2584if (LIST_FIRST(bp-b_dep) != NULL bioops.io_complete) (gdb) p bp $6 = (struct buf *) 0xc1550c18 (gdb) p *bp $7 = {b_hash = {le_next = 0x0, le_prev = 0x0}, b_vnbufs = {tqe_next = 0x0, tqe_prev = 0x0}, b_freelist = {tqe_next = 0x0, tqe_prev = 0x0}, b_act = { tqe_next = 0x0, tqe_prev = 0xc13e4400}, b_flags = 134218244, b_qindex = 0, b_unused1 = 0 '\000', b_xflags = 0 '\000', b_lock = {lk_interlock = { lock_data = 0}, lk_flags = 1024, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 1, lk_prio = 20, lk_wmesg = 0x0, lk_timo = 0, lk_lockholder = 5}, b_error = 0, b_bufsize = 8192, b_bcount = 8192, b_resid = 0, b_dev = 0xc1434d00, b_data = 0xc1954000 "íA\003", b_kvabase = 0x0, b_kvasize = 0, b_lblkno = 0, b_blkno = 0, b_offset = 0, b_iodone = 0, b_iodone_chain = 0x0, b_vp = 0xd5725980, b_dirtyoff = 0, b_dirtyend = 0, b_rcred = 0x, b_wcred = 0x0, b_pblkno = 1389417, b_saveaddr = 0x0, b_driver1 = 0x0, b_driver2 = 0x0, b_caller1 = 0x0, b_caller2 = 0x0, b_pager = {pg_spc = 0x0, pg_reqpage = 0}, b_cluster = { cluster_head = {tqh_first = 0x0, tqh_last = 0x0}, cluster_entry = { tqe_next = 0x0, tqe_prev = 0x0}}, b_pages = {0x0 repeats 32 times}, b_npages = 0, b_dep = {lh_first = 0x0}, b_chain = {parent = 0x0, count = 0}} (gdb) OK, this is indeed the same problem that Bernd has been seeing. B_CALL should be set, but so should b_iodone. In the header B_CALL isn't set, but that's not surprising in view of the code. What I don't understand is where b_iodone went. I *always* set b_iodone, and I've been through the entire source tree looking for where it might get reset. The only place is in geteblk, when allocating a fresh buffer. I can't make head or tail of it. Try these patches. They won't fix the problem, but they may help localize it: Index: kern/vfs_bio.c === RCS file: /home/ncvs/src/sys/kern/vfs_bio.c,v retrieving revision 1.227 diff -w -u -r1.227 vfs_bio.c --- vfs_bio.c 1999/08/28 00:46:23 1.227 +++ vfs_bio.c 1999/08/30 01:31:00 @@ -2576,6 +2576,8 @@ /* call optional completion function if requested */ if (bp-b_flags B_CALL) { + if (bp-b_iodone == NULL) + Debugger ("biodone"); bp-b_flags = ~B_CALL; (*bp-b_iodone) (bp); splx(s); Index: dev/vinum/vinumrequest.c === RCS file: /home/ncvs/src/sys/dev/vinum/vinumrequest.c,v retrieving revision 1.35 diff -w -u -r1.35 vinumrequest.c --- vinumrequest.c 1999/08/28 00:42:42 1.35 +++ vinumrequest.c 1999/08/30 01:26:58 @@ -396,6 +390,9 @@ if (debug DEBUG_LASTREQS) logrq(loginfo_rqe, (union rqinfou) rqe, rq-bp); #endif + if ((rqe-b.b_flags B_CALL) +(rqe-b.b_iodone == NULL) ) + Debugger ("launch_requests"); /* fire off the request */ BUF_STRATEGY(rqe-b, 0); } You could also enable some of Vinum's internal logging: # vinum debug 324 This will log all requests in an internal buffer. With the .gdbinit files in /sys/modules/vinum, you can display them with the
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
:(kgdb) target remote /dev/cuaa2 :Remote debugging using /dev/cuaa2 :0x0 in ?? () :(kgdb) bt :#0 0x0 in ?? () :#1 0xc017228f in biodone (bp=0xc09ebd80) at ../../kern/vfs_bio.c:2580 :#2 0xc0123db6 in dadone (periph=0xc0882c80, done_ccb=0xc09bd200) at :../../cam/scsi/scsi_da.c:1294 :#3 0xc011fa7b in camisr (queue=0xc02726b4) at ../../cam/cam_xpt.c:6141 :#4 0xc011f88d in swi_cambio () at ../../cam/cam_xpt.c:6048 :#5 0xc020db30 in splz_swi () :(kgdb) frame 1 :#1 0xc017228f in biodone (bp=0xc09ebd80) at ../../kern/vfs_bio.c:2580 :2580 ../../kern/vfs_bio.c: No such file or directory. :(kgdb) print bp :$1 = (struct buf *) 0xc09ebd80 :(kgdb) print *bp :$2 = { : b_hash = { :le_next = 0x0, :le_prev = 0x0 : }, : b_vnbufs = { :... :(kgdb) :-- :B.Walter COSMO-Project http://www.cosmo-project.de This is definitely a pbuf. Did you apply the patches Greg emailed? They will panic the machine earlier while it is still in the correct stack frame, allowing Greg to track down where the I/O initiation came from. -Matt Matthew Dillon [EMAIL PROTECTED] (patch and comments from Greg): Index: kern/vfs_bio.c === RCS file: /home/ncvs/src/sys/kern/vfs_bio.c,v retrieving revision 1.227 diff -w -u -r1.227 vfs_bio.c --- vfs_bio.c 1999/08/28 00:46:23 1.227 +++ vfs_bio.c 1999/08/30 01:31:00 @@ -2576,6 +2576,8 @@ /* call optional completion function if requested */ if (bp-b_flags B_CALL) { + if (bp-b_iodone == NULL) + Debugger ("biodone"); bp-b_flags = ~B_CALL; (*bp-b_iodone) (bp); splx(s); Index: dev/vinum/vinumrequest.c === RCS file: /home/ncvs/src/sys/dev/vinum/vinumrequest.c,v retrieving revision 1.35 diff -w -u -r1.35 vinumrequest.c --- vinumrequest.c 1999/08/28 00:42:42 1.35 +++ vinumrequest.c 1999/08/30 01:26:58 @@ -396,6 +390,9 @@ if (debug DEBUG_LASTREQS) logrq(loginfo_rqe, (union rqinfou) rqe, rq-bp); #endif + if ((rqe-b.b_flags B_CALL) +(rqe-b.b_iodone == NULL) ) + Debugger ("launch_requests"); /* fire off the request */ BUF_STRATEGY(rqe-b, 0); } You could also enable some of Vinum's internal logging: # vinum debug 324 This will log all requests in an internal buffer. With the .gdbinit files in /sys/modules/vinum, you can display them with the gdb command 'rrqi'. Alternatively, if I can connect to the debug console, I'll look for myself. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
:Greg's debugging this on the machine here at the moment, which is :crashing the same way in what appears to be the same place. : :If anyone else wants to take a crack it, the magic vinum debug number :needed is 328 and not 324. : :So far Greg's discovered that the field is correctly set a little while :before the crash, and then becomes NULL for no apparent reason. Looks :like a really nasty bug. : :The nice thing is this loaner machine has no real job to do besides :crashing, so it can crash as often as required. :) : : : -- Parag I looked at the vinum code a little. I would look for possible situations where the bp may be re-issued as an I/O before actually completing a prior I/O. This could cause the iodone field to be cleared in the middle of an I/O. I don't see anything specific in the code because I don't really understand it yet. That's the best possibility that I can come up with. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
On Sunday, 29 August 1999 at 23:15:47 -0700, Matthew Dillon wrote: Greg's debugging this on the machine here at the moment, which is crashing the same way in what appears to be the same place. If anyone else wants to take a crack it, the magic vinum debug number needed is 328 and not 324. So far Greg's discovered that the field is correctly set a little while before the crash, and then becomes NULL for no apparent reason. Looks like a really nasty bug. The nice thing is this loaner machine has no real job to do besides crashing, so it can crash as often as required. :) I looked at the vinum code a little. I would look for possible situations where the bp may be re-issued as an I/O before actually completing a prior I/O. This could cause the iodone field to be cleared in the middle of an I/O. I don't see anything specific in the code because I don't really understand it yet. That's the best possibility that I can come up with. Well, I don't do things like that. The only place I reissue a command is in the iodone routine (complete_rqe or complete_raid5_write; the latter in this case). Greg -- See complete headers for address, home page and phone numbers finger [EMAIL PROTECTED] for PGP public key To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
On Monday, 30 August 1999 at 7:53:12 +0200, Bernd Walter wrote: On Sun, Aug 29, 1999 at 04:48:32PM -0700, Matthew Dillon wrote: : :How similar? The trap above is extremely bad; it looks like a return :on a corrupted stack or a jump through a null function vector. : :Make very sure that your vinum kld is in sync with your kernel. This looks like an indirect call through a NULL function pointer. In my case it is a call to bp-b_iodone in kern/vfs_bio.c:2580 which is 0 :( Yes, this is the same thing. Until Parag came along, I was beginning to think it was a problem with your hardware :-( Is there any way for me to look at this? Do you have an IDE disk on your machine that you could dump to? Greg -- See complete headers for address, home page and phone numbers finger [EMAIL PROTECTED] for PGP public key To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
On Sunday, 29 August 1999 at 22:59:22 -0700, Matthew Dillon wrote: (kgdb) target remote /dev/cuaa2 Remote debugging using /dev/cuaa2 0x0 in ?? () (kgdb) bt #0 0x0 in ?? () #1 0xc017228f in biodone (bp=0xc09ebd80) at ../../kern/vfs_bio.c:2580 #2 0xc0123db6 in dadone (periph=0xc0882c80, done_ccb=0xc09bd200) at ../../cam/scsi/scsi_da.c:1294 #3 0xc011fa7b in camisr (queue=0xc02726b4) at ../../cam/cam_xpt.c:6141 #4 0xc011f88d in swi_cambio () at ../../cam/cam_xpt.c:6048 #5 0xc020db30 in splz_swi () (kgdb) frame 1 #1 0xc017228f in biodone (bp=0xc09ebd80) at ../../kern/vfs_bio.c:2580 2580 ../../kern/vfs_bio.c: No such file or directory. (kgdb) print bp $1 = (struct buf *) 0xc09ebd80 (kgdb) print *bp $2 = { b_hash = { le_next = 0x0, le_prev = 0x0 }, b_vnbufs = { ... (kgdb) This is definitely a pbuf. What's a pbuf? Did you apply the patches Greg emailed? They will panic the machine earlier while it is still in the correct stack frame, allowing Greg to track down where the I/O initiation came from. Well, in fact they didn't do that. The only one that I hit was in biodone, which is effectively the same thing as the panic. But they did show that it wasn't like that when it left vinum. Index: kern/vfs_bio.c === RCS file: /home/ncvs/src/sys/kern/vfs_bio.c,v retrieving revision 1.227 diff -w -u -r1.227 vfs_bio.c --- vfs_bio.c 1999/08/28 00:46:23 1.227 +++ vfs_bio.c 1999/08/30 01:31:00 @@ -2576,6 +2576,8 @@ /* call optional completion function if requested */ if (bp-b_flags B_CALL) { + if (bp-b_iodone == NULL) + Debugger ("biodone"); bp-b_flags = ~B_CALL; (*bp-b_iodone) (bp); splx(s); Index: dev/vinum/vinumrequest.c === RCS file: /home/ncvs/src/sys/dev/vinum/vinumrequest.c,v retrieving revision 1.35 diff -w -u -r1.35 vinumrequest.c --- vinumrequest.c 1999/08/28 00:42:42 1.35 +++ vinumrequest.c 1999/08/30 01:26:58 @@ -396,6 +390,9 @@ if (debug DEBUG_LASTREQS) logrq(loginfo_rqe, (union rqinfou) rqe, rq-bp); #endif + if ((rqe-b.b_flags B_CALL) +(rqe-b.b_iodone == NULL) ) + Debugger ("launch_requests"); /* fire off the request */ BUF_STRATEGY(rqe-b, 0); } You could also enable some of Vinum's internal logging: # vinum debug 324 Note that this should read 328. 324 will always land you in the Debugger (a feature, not a bug :-). Greg -- See complete headers for address, home page and phone numbers finger [EMAIL PROTECTED] for PGP public key To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
: }, : b_vnbufs = { : ... : (kgdb) : : This is definitely a pbuf. : :What's a pbuf? Sorry, it isn't a pbuf. Hmm. You are allocating and initializing a struct buf yourself? That could cause long term problems but is neither here nor there at the moment, we'll deal with it later. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
On Sunday, 29 August 1999 at 23:36:24 -0700, Matthew Dillon wrote: }, b_vnbufs = { ... (kgdb) This is definitely a pbuf. What's a pbuf? Sorry, it isn't a pbuf. OK. But what is a pbuf? I've never heard that term before. Hmm. You are allocating and initializing a struct buf yourself? Yup. I need more information in the header, and at the time I did it I was told the b_driver* fields would go away. That could cause long term problems but is neither here nor there at the moment, we'll deal with it later. I discussed it with Kirk, who wasn't too concerned. Do you have other issues? Greg -- See complete headers for address, home page and phone numbers finger [EMAIL PROTECTED] for PGP public key To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
On Monday, 30 August 1999 at 8:06:03 +0200, Bernd Walter wrote: On Sun, Aug 29, 1999 at 10:59:22PM -0700, Matthew Dillon wrote: This is definitely a pbuf. Did you apply the patches Greg emailed? Not yet. I will do it today. Wait a while. I'll send you another couple as well. Greg -- See complete headers for address, home page and phone numbers finger [EMAIL PROTECTED] for PGP public key To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: 4.0-CURRENT SMP crash with vinum raid-5 and softupdates
: : What's a pbuf? : : Sorry, it isn't a pbuf. : :OK. But what is a pbuf? I've never heard that term before. A pbuf is a 'physical buffer'. Specifically, it is a struct buf structure used by low level device drivers to issue their own I/O. pbuf's are used by a number of modules. For example, they are used by the VFS clustering code to create a 'super buffer' that encompasses a number of smaller filesystem buffers. They are also used by the swapper to initiate low level disk I/O, and the VN device for same. : That could cause long term problems but is neither here nor : there at the moment, we'll deal with it later. : :I discussed it with Kirk, who wasn't too concerned. Do you have other :issues? : :Greg Naw, when we finally fix the struct buf's we'll deal with everything that uses them. As long as you do not special case structural fields (use them in a manner for which they were not designed), there shouldn't be any problem. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message