Re: gmirror slice insertion, FAILURE - READ_DMA status=51READY, DSC, ERROR

2008-11-01 Thread CyberLeo Kitsana
Thomas Sparrevohn wrote:
 The error occured after I had the disk for a couple of days - WHat puzzled me 
 was that the drive
 did not do it automatically 

Hard disks will not map uncorrectable bad sectors on read automatically,
as it no longer knows what the contents of that sector should be. In
this instance, the sector is usually remapped during a write.

Given the symptoms of the problem described above, it looks like this
uncorrectable sector is located in a portion of the disk that isn't
touched by FreeBSD's newfs or installation procedure, and would never
have a chance to be written to and corrected. Then, when the mirror sync
occurs (which copies every block verbatim, regardless of whether it's in
use or not) it's choking on that sector and locking up the disk, thus
freezing the OS.

One thing to try prior to RMAing the disk is to fill the entire disk
with zeroes (dd if=/dev/zero of=/dev/ad6 bs=131072 or similar) to give
its firmware a chance to remap all flakey sectors, and rewrite all ECC
information. I do this with every new or freshly acquired disk that's
guaranteed to be empty, to ensure that no surprise errors bite me later
on, as well as to make sure no previous data hangs around.

-- 
Fuzzy love,
-CyberLeo
Technical Administrator
CyberLeo.Net Webhosting
http://www.CyberLeo.Net
[EMAIL PROTECTED]

Furry Peace! - http://.fur.com/peace/
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: gmirror slice insertion, FAILURE - READ_DMA status=51READY, DSC, ERROR

2008-10-31 Thread Thomas Sparrevohn
On Wednesday 29 October 2008 10:04:39 Jeremy Chadwick wrote:
 On Wed, Oct 29, 2008 at 02:00:21AM -0700, Carl wrote:
  Jeremy Chadwick wrote:
  Seagate chooses to encode some raw data for some SMART attributes in a
  custom format.  The format is not publicly documented.  This is why you
  have to go off of the adjusted values shown in VALUE/WORST/THRESH.
  How am I supposed to know all of this?!  You aren't -- it comes with
  experience.
 
  And yet my failing drive's VALUE numbers are still all above their  
  THRESH values, despite it being bad enough to cripple the system. One  
  might argue those threshold values leave something to be desired.
 
 I'd urge you to file complaint(s) with drive manufacturers, as they're
 the ones who decide the values.  Thresholds are not defined per the
 ATA-ATAPI specification, so technically they can pick whatever value
 they want.  This is exactly why you'll encounter people screaming SMART
 is worthless, the drive is already dead by the time the overall SMART
 health check fails!
 
 If you go this route, please CC me, as I'd be quite to see what
 manufacturers have to say.
 

Just a saw note - I saw the same problem with a hitachi disk - I ran a vendor 
diagnostics tool
that I found on their home page and it rebuild the bad sector map and the 
problem went away 

The error occured after I had the disk for a couple of days - WHat puzzled me 
was that the drive
did not do it automatically 
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: gmirror slice insertion, FAILURE - READ_DMA status=51READY, DSC, ERROR

2008-10-29 Thread Carl

Jeremy Chadwick wrote:

Seagate chooses to encode some raw data for some SMART attributes in a
custom format.  The format is not publicly documented.  This is why you
have to go off of the adjusted values shown in VALUE/WORST/THRESH.
How am I supposed to know all of this?!  You aren't -- it comes with
experience.


And yet my failing drive's VALUE numbers are still all above their 
THRESH values, despite it being bad enough to cripple the system. One 
might argue those threshold values leave something to be desired.


Is there anything I should know about this model of hard disk with  
regards to being known for problems? Also, is there a good test I can  
perform to hopefully flush out any problems before I put this thing into  
service?


I'm confused: what gives you the impression there's a problem with
*this model* of hard disk?  I've seen no evidence presented that
indicates such.  What makes you ask that question?


I don't have such an impression, thus far. In fact, Seagate drives have 
always been good to me prior to this. It's only a precautionary question 
because it's better to ask now than after I've committed a lot of real 
data and time to it and put it all into service.



Let's take a look at the SMART data.


# smartctl -a /dev/ad4

ID# ATTRIBUTE_NAME  FLAG VALUE WORST THRESH TYPE UPDATED  
WHEN_FAILED   RAW_VALUE

...

198 Offline_Uncorrectable   0x0010   100   100   000Old_age  Offline
-   0

...


To get an update on Attribute 198, you'd need to run a short offline
test (smartctl -t short /dev/ad4).  You can safely do this while
the disk is in use; don't let the word offline make you think the
disk disappears.  You can watch the status using smartctl -a, and
once its finished, you can compare the old value to the new.  I'm
willing to bet it remains zero.


I ran that test on both drives. ad6 failed immediately at 90% with a 
read failure - not surprising. ad4 completed without error and no 
change in it's values, just as you predicted.



# smartctl -a /dev/ad6

ID# ATTRIBUTE_NAME  FLAG VALUE WORST THRESH TYPE UPDATED  
WHEN_FAILED   RAW_VALUE

...

  5 Reallocated_Sector_Ct   0x0033   100   100   036Pre-fail Always 
-   2

...

 10 Spin_Retry_Count0x0013   100   100   097Pre-fail Always 
-   1

...

187 Reported_Uncorrect  0x0032   098   098   000Old_age  Always 
-   2

...

197 Current_Pending_Sector  0x0012   100   100   000Old_age  Always 
-   2
198 Offline_Uncorrectable   0x0010   100   100   000Old_age  Offline
-   2

...


And here we see the core of the problem.  :-)



Advice is simple: replace this hard disk.



Hope this helps.


It definitely did, Jeremy. Your explanations were most helpful. Thanks!

Carl / K0802647

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: gmirror slice insertion, FAILURE - READ_DMA status=51READY, DSC, ERROR

2008-10-29 Thread Jeremy Chadwick
On Wed, Oct 29, 2008 at 02:00:21AM -0700, Carl wrote:
 Jeremy Chadwick wrote:
 Seagate chooses to encode some raw data for some SMART attributes in a
 custom format.  The format is not publicly documented.  This is why you
 have to go off of the adjusted values shown in VALUE/WORST/THRESH.
 How am I supposed to know all of this?!  You aren't -- it comes with
 experience.

 And yet my failing drive's VALUE numbers are still all above their  
 THRESH values, despite it being bad enough to cripple the system. One  
 might argue those threshold values leave something to be desired.

I'd urge you to file complaint(s) with drive manufacturers, as they're
the ones who decide the values.  Thresholds are not defined per the
ATA-ATAPI specification, so technically they can pick whatever value
they want.  This is exactly why you'll encounter people screaming SMART
is worthless, the drive is already dead by the time the overall SMART
health check fails!

If you go this route, please CC me, as I'd be quite to see what
manufacturers have to say.

-- 
| Jeremy Chadwickjdc at parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: gmirror slice insertion, FAILURE - READ_DMA status=51READY, DSC, ERROR

2008-10-29 Thread Jeremy Chadwick
On Tue, Oct 28, 2008 at 08:41:31PM -0700, Carl wrote:
 Jeremy Chadwick said:
 ad6: FAILURE - READ_DMA status=51READY,DSC,ERROR   
 error=40UNCORRECTABLE LBA=134802751

 Are you sure you don't have a bad hard disk?  This looks to be like a
 classic block/sector failure.

 I hadn't realized that a bad block would manifest itself with a message  
 about DMA. Seems like such semantics would be a little obscure to most  
 users, apparently including me.

Do not let the term DMA confuse you -- the operation was a read
operation, and DMA is used to do the transfer of data between
disk/controller/local memory.  You might see things like READ_DMA48
and WRITE_DMA48, which just indicate that 48-bit LBA addressing mode
is in use when attempting the operation.

For sake of comparison, you should see what Linux and Solaris do.  For
example, when a disk falls off the bus (silently) on a Linux machine
using ext3fs, all I've ever seen is continual spewing of ext3fs journal
errors on the console -- absolutely no indication that the disk itself
has actually fallen off the bus.  With SCSI disks under Solaris, the
level of detail you get is perfect -- it's very easy to determine what
happened.  But in the case of ATA disks, you get more or less something
that looks similar to FreeBSD.

If you have complaints about the formatting of the output, I would
recommend filing a PR for it, or bringing it up with Soren Schmidt
([EMAIL PROTECTED]), author of the ata(4) layer.  I will agree with you
that some more coherent error messages would be useful.

 So you're saying that the *exact* same READ_DMA error, at the *exact*
 same LBA, is reported on ad4?  If so, that's very bizarre.

 No, perhaps I wasn't clear enough. Both instances were on ad6, so far.

Then that makes ad6, or something specific to ad6, the culprit.

 Can you please provide the output from the following commands?

 See end of message. Let me know if you then want more (in- or out-of-band).

 Having now installed smartmontools, you can see below that I ran it for  
 both ad4 and ad6. Sure enough, ad6 has logged 2 READ DMA errors - does  
 that make this a definitive bad disk then?

I'll have to look at the output.  See below.

 Should I not be worried about ad4 too? Those Raw_Read_Error_Rate and  
 Seek_Error_Rate numbers should be zero or very close to it, shouldn't  
 they? I don't know how to interpret what I'm seeing in that output, so  
 I'd appreciate any insight. Should I be returning both disks for  
 warranty claims (they're both very recently purchased)?

As you've admitted, the problem is that most people don't know how to
interpret SMART data, and start freaking out over things which are
normal.  People focus on the RAW values, which for many attributes is
the wrong thing to look at.  For example, on Seagate disks, a insanely
high Raw_Read_Error_Rate and Seek_Error_Rate means absolutely nothing;
it's normal.  But with another vendor, it might actually be accurate.
Welcome to one of the problems with SMART: the specification does not
state what format the raw data must be in.

Seagate chooses to encode some raw data for some SMART attributes in a
custom format.  The format is not publicly documented.  This is why you
have to go off of the adjusted values shown in VALUE/WORST/THRESH.
How am I supposed to know all of this?!  You aren't -- it comes with
experience.

 Is there anything I should know about this model of hard disk with  
 regards to being known for problems? Also, is there a good test I can  
 perform to hopefully flush out any problems before I put this thing into  
 service?

I'm confused: what gives you the impression there's a problem with
*this model* of hard disk?  I've seen no evidence presented that
indicates such.  What makes you ask that question?

None of us here work at Seagate, so even if there was a known problem
with this specific model of disk, we wouldn't know.  For all we know,
there could be little 3mm tall terrorists dancing on the platters, ready
to leap out at any moment and stab us!  :-)

Please keep something in mind: just because you have brand new hard
disks *does not* guarantee they're free of errors.  I have seen hundreds
of brand new hard disks fail right out of the box, including SCSI
disks (which people, for some reason, think are less likely to have
this problem simply because they cost more money).  I deal with this
situation on a daily basis at work, believe it or not.

 # vmstat -i

Interrupts look fine; I was looking for anything that might indicate an
absurdly high rate.

atacontrol cap output looks fine too, nothing weird or out of the
ordinary (I wasn't expecting anything to show up here, but I did want to
get an idea if the disks were truly SATA300 or not).

Let's take a look at the SMART data.

 # smartctl -a /dev/ad4

 ID# ATTRIBUTE_NAME  FLAG VALUE WORST THRESH TYPE UPDATED  
 WHEN_FAILED   RAW_VALUE
   1 Raw_Read_Error_Rate 0x000f   117   099   006Pre-fail Always   
   -   

Re: gmirror slice insertion, FAILURE - READ_DMA status=51READY, DSC, ERROR

2008-10-28 Thread Wojciech Puchar
so good. Began Ralf's procedure for inserting ad4s1 into mirror/gm0. The 
synchronization began and reached 6% when this little horror appeared:


ad6: FAILURE - READ_DMA status=51READY,DSC,ERROR error=40UNCORRECTABLE 
LBA=134802751
GEOM_MIRROR: Request failed (error=5). ad6s1[READ(offset=69018976256, 
length=131072)]
GEOM_MIRROR: Synchronization request failed (error=5). 
mirror/gm0[READ(offset=69018976256, length=131072)]




your disk failed. (uncorrectable error)

assuming you eliminated other causes like drives overheating, cabling 
problem (don't think so) etc.:



boot from some kind of live CD, then make another mirror (single disk now) 
on other drive, then do


dd if=/dev/ad6s1 of=/dev/mirror/newmirror bs=2k conv=noerror,sync

i intentionally did bs=2k instead of larger, to minimize amount of lost 
data.


then change your system to boot from newmirror, take out /dev/ad6 and have 
it replaced on warranty (or buy new), put new ad6, insert it to the 
mirror.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: gmirror slice insertion, FAILURE - READ_DMA status=51READY, DSC, ERROR

2008-10-28 Thread Wojciech Puchar

error=40UNCORRECTABLE LBA=134802751


Are you sure you don't have a bad hard disk?  This looks to be like a
classic block/sector failure.  This does not appear to be the infamous
famous DMA timeout problem, especially if this is the only error
you're getting.


he can temporarity boot with hw.ata.ata_dma=0

but i think his drive failed.




I reinstalled FB7 to ad4, redid the /boot.config modification to make
ad6/gm0 bootable again and retried the insertion of ad4 into gm0. Exact
same error messages at exactly the same point with same consequences.



so IT IS FAILED drive!
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: gmirror slice insertion, FAILURE - READ_DMA status=51READY, DSC, ERROR

2008-10-28 Thread Jeremy Chadwick
On Tue, Oct 28, 2008 at 12:04:49PM +0100, Wojciech Puchar wrote:
 error=40UNCORRECTABLE LBA=134802751

 Are you sure you don't have a bad hard disk?  This looks to be like a
 classic block/sector failure.  This does not appear to be the infamous
 famous DMA timeout problem, especially if this is the only error
 you're getting.

 he can temporarity boot with hw.ata.ata_dma=0

They're SATA disks, so this won't do anything sadly.

-- 
| Jeremy Chadwickjdc at parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: gmirror slice insertion, FAILURE - READ_DMA status=51READY, DSC, ERROR

2008-10-28 Thread Carl

Jeremy Chadwick said:
ad6: FAILURE - READ_DMA status=51READY,DSC,ERROR  
error=40UNCORRECTABLE LBA=134802751


Are you sure you don't have a bad hard disk?  This looks to be like a
classic block/sector failure.


I hadn't realized that a bad block would manifest itself with a message 
about DMA. Seems like such semantics would be a little obscure to most 
users, apparently including me.



So you're saying that the *exact* same READ_DMA error, at the *exact*
same LBA, is reported on ad4?  If so, that's very bizarre.


No, perhaps I wasn't clear enough. Both instances were on ad6, so far.


Can you please provide the output from the following commands?


See end of message. Let me know if you then want more (in- or out-of-band).

Having now installed smartmontools, you can see below that I ran it for 
both ad4 and ad6. Sure enough, ad6 has logged 2 READ DMA errors - does 
that make this a definitive bad disk then?


Should I not be worried about ad4 too? Those Raw_Read_Error_Rate and 
Seek_Error_Rate numbers should be zero or very close to it, shouldn't 
they? I don't know how to interpret what I'm seeing in that output, so 
I'd appreciate any insight. Should I be returning both disks for 
warranty claims (they're both very recently purchased)?


Wojciech Puchar said:
boot from some kind of live CD, then make another mirror (single disk now) 
on other drive, then do


dd if=/dev/ad6s1 of=/dev/mirror/newmirror bs=2k conv=noerror,sync

i intentionally did bs=2k instead of larger, to minimize amount of lost 
data.


then change your system to boot from newmirror, take out /dev/ad6 and have 
it replaced on warranty (or buy new), put new ad6, insert it to the 
mirror.


I think you're describing a method to help me save as much data from ad6 
as possible. Fortunately, this is all about constructing a new system, 
so there's no data yet to lose.


Is there anything I should know about this model of hard disk with 
regards to being known for problems? Also, is there a good test I can 
perform to hopefully flush out any problems before I put this thing into 
service?


Carl / K0802647

 Additional Information 

# vmstat -i
interrupt  total   rate
irq1: atkbd0   4  0
irq4: sio0125724 16
irq19: uhci3   5  0
irq21: uhci1+ 478364 63
irq23: uhci2 ehci1 1  0
cpu0: timer 14517071   1923
irq256: em0   109568 14
cpu1: timer 14514956   1922
Total   29745693   3940

# atacontrol list | grep -v no device present
ATA channel 0:
ATA channel 1:
ATA channel 2:
Master:  ad4 ST31000340AS/SD15 Serial ATA II
ATA channel 3:
Master:  ad6 ST31000340AS/SD15 Serial ATA II
ATA channel 4:
Master: acd0 HL-DT-ST DVDRAM GH20NS10/EL00 Serial ATA v1.0
ATA channel 5:
ATA channel 6:
ATA channel 7:

# atacontrol cap ad4

Protocol  Serial ATA II
device model  ST31000340AS
serial number xxxH
firmware revision SD15
cylinders 16383
heads 16
sectors/track 63
lba supported 268435455 sectors
lba48 supported   1953525168 sectors
dma supported
overlap not supported

Feature  Support  EnableValue   Vendor
write cacheyes  yes
read ahead yes  yes
Native Command Queuing (NCQ)   yes   -  31/0x1F
Tagged Command Queuing (TCQ)   no   no  31/0x1F
SMART  yes  yes
microcode download yes  yes
security   yes  no
power management   yes  yes
advanced power management  no   no  65278/0xFEFE
automatic acoustic management  no   no  0/0x00  254/0xFE

# atacontrol cap ad6

Protocol  Serial ATA II
device model  ST31000340AS
serial number xxxA
firmware revision SD15
cylinders 16383
heads 16
sectors/track 63
lba supported 268435455 sectors
lba48 supported   1953525168 sectors
dma supported
overlap not supported

Feature  Support  EnableValue   Vendor
write cacheyes  yes
read ahead yes  yes
Native Command Queuing (NCQ)   yes   -  31/0x1F
Tagged Command Queuing (TCQ)   no   no  31/0x1F
SMART  yes  yes
microcode download yes  yes
security   yes  no
power management   yes  yes
advanced power management  no   no  65278/0xFEFE
automatic acoustic management  no   no  0/0x00  254/0xFE

# smartctl -a /dev/ad4
smartctl version 5.38 [i386-portbld-freebsd7.0] Copyright (C) 2002-8 

gmirror slice insertion, FAILURE - READ_DMA status=51READY, DSC, ERROR

2008-10-27 Thread Carl Voth
I'm setting up a dual-disk server and am trying to bring it up with 
gmirror and gjournal. One slice per disk, the goal being to create a 
single mirror from said slices with some of the partitions journaled. 
Installed FreeBSD-7.0RELEASE to ad4, then used technique from here to 
create single-disk mirror/gm0 on ad6:


  http://people.freebsd.org/~rse/mirror/

Modified ad4s1a /boot.config to pass control to boot stage 3 on ad6. So 
far, so good. Began Ralf's procedure for inserting ad4s1 into 
mirror/gm0. The synchronization began and reached 6% when this little 
horror appeared:


ad6: FAILURE - READ_DMA status=51READY,DSC,ERROR 
error=40UNCORRECTABLE LBA=134802751
GEOM_MIRROR: Request failed (error=5). ad6s1[READ(offset=69018976256, 
length=131072)]
GEOM_MIRROR: Synchronization request failed (error=5). 
mirror/gm0[READ(offset=69018976256, length=131072)]


After that, nothing. System unresponsive. Perhaps needless to say, the 
system also becomes unbootable because the whole point here was to nuke 
ad4 as part of inserting it into mirror/gm0.


I reinstalled FB7 to ad4, redid the /boot.config modification to make 
ad6/gm0 bootable again and retried the insertion of ad4 into gm0. Exact 
same error messages at exactly the same point with same consequences. 
Now, I see that other folks are having unexplained DMA problems too, 
albeit in different contexts. What should I be concluding here? Those 
other folks don't seem to be concluding it's bad drives. If there were 
bad sectors, I'd get different error messages, yes?


FWIW, I'm using gjournal on 3 partitions in mirror/gm0.

Here's my server's parts list:
- Intel S3210SHLC Motherboard.
- Kingston KVR800D2E5/2GI 2GB DRAM (x2).
- Intel BX80570E3110 Dual-Core Xeon E3110, 3 Ghz 6MB L2 Cache, LGA775.
- Seagate ST31000340AS Barracuda 7200.11, 1TB, SATA (x2).
- LG GH20NS10 Internal Super-Multi SecurDisc 20X SATA DVD Rewriter.
- Antec 4U22EPS650XR case (NeoPower 650W PSU).

Carl / K0802647
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: gmirror slice insertion, FAILURE - READ_DMA status=51READY, DSC, ERROR

2008-10-27 Thread Jeremy Chadwick
On Mon, Oct 27, 2008 at 06:56:24PM -0700, Carl Voth wrote:
 I'm setting up a dual-disk server and am trying to bring it up with  
 gmirror and gjournal. One slice per disk, the goal being to create a  
 single mirror from said slices with some of the partitions journaled.  
 Installed FreeBSD-7.0RELEASE to ad4, then used technique from here to  
 create single-disk mirror/gm0 on ad6:

   http://people.freebsd.org/~rse/mirror/

 Modified ad4s1a /boot.config to pass control to boot stage 3 on ad6. So  
 far, so good. Began Ralf's procedure for inserting ad4s1 into  
 mirror/gm0. The synchronization began and reached 6% when this little  
 horror appeared:

 ad6: FAILURE - READ_DMA status=51READY,DSC,ERROR  
 error=40UNCORRECTABLE LBA=134802751

Are you sure you don't have a bad hard disk?  This looks to be like a
classic block/sector failure.  This does not appear to be the infamous
famous DMA timeout problem, especially if this is the only error
you're getting.

 I reinstalled FB7 to ad4, redid the /boot.config modification to make  
 ad6/gm0 bootable again and retried the insertion of ad4 into gm0. Exact  
 same error messages at exactly the same point with same consequences.  

So you're saying that the *exact* same READ_DMA error, at the *exact*
same LBA, is reported on ad4?  If so, that's very bizarre.

 Now, I see that other folks are having unexplained DMA problems too,  
 albeit in different contexts. What should I be concluding here? Those  
 other folks don't seem to be concluding it's bad drives. If there were  
 bad sectors, I'd get different error messages, yes?

The error=40UNCORRECTABLE part of what you're seeing seems to imply
there's an uncorrectable read transaction that's happened.  What other
people see are DMA timeouts, but no actual sign of uncorrectable errors.

The problem with the DMA timeout issue is that it manifests itself in
hundreds of different ways.  Each case so far has to be handled on an
individual basis.

 FWIW, I'm using gjournal on 3 partitions in mirror/gm0.

 Here's my server's parts list:
 - Seagate ST31000340AS Barracuda 7200.11, 1TB, SATA (x2).

Can you please provide the output from the following commands?

dmesg
vmstat -i
atacontrol list
atacontrol cap ad4
atacontrol cap ad6
smartctl -a /dev/ad4
smartctl -a /dev/ad6

Thanks.

-- 
| Jeremy Chadwickjdc at parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]