Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Sid Spry]

On Tue, Jun 23, 2020, at 12:20 PM, Rich Freeman wrote:
> On Tue, Jun 23, 2020 at 12:14 PM Sid Spry  wrote:
> >
> > So if I'm understanding properly most drive firmware won't let you
> > operate the device in an append-only mode?
> 
> So, there are several types of SMR drives.
> 
> There are host-managed, drive-managed, and then hybrid devices that
> default to drive-managed for compatibility reasons but the host can
> send them a command to take full control so that it is the same as
> host-managed.
> 
> A host-managed drive just does what the host tells it to.  If the host
> tells it to do a write that obliterates some other data on the disk,
> the drive just does it, and it is the job of the host
> OS/filesystem/application to make sure that they protect any data they
> care about.  At the drive level these perform identically to CMR
> because they just seek and write like any other drive.  At the
> application level these could perform differently since the
> application might end up having to work around the drive.  However,
> these drives are generally chosen for applications where this is not a
> big problem or where the problems can be efficiently mitigated.
> 
> A drive-managed drive just looks like a regular drive to the host, and
> it ends up having to do a lot of read-before-rewrite operations
> because the host is treating it like it is CMR but the drive has to
> guarantee that nothing gets lost.  A drive-managed disk has no way to
> operate in "append-only" mode.  I'm not an expert in ATA but I believe
> disks are just given an LBA and a set of data to write.  Without
> support for TRIM the drive has no way to know if it is safe to
> overwrite nearby cylinders, which means it has to preserve them.
> 

Yeah, this is what I was wondering. It looks like there are devices that
do management that keeps you from using them at their full
performance.

> The biggest problem is that the vendors were trying to conceal the
> nature of the drives.  If they advertised TRIM support it would be
> pretty obvious they were SMR.
> 

It looks like I was right then. Maybe the market will settle soon and I
will be able to buy properly marked parts. It's a good thing I stumbled
into this, I was going to be buying more storage shortly.

> > If any do I suspect
> > NILFS (https://en.wikipedia.org/wiki/NILFS) may be a good choice:
> >
> > "NILFS is a log-structured filesystem, in that the storage medium is treated
> > like a circular buffer and new blocks are always written to the end. [...]"
> >
> 
> On a host-managed disk this would perform the same as on a CMR disk,
> with the possible exception of any fixed metadata (I haven't read the
> gory details on the filesystem).  If it has no fixed metadata (without
> surrounding unused tracks) then it would have no issues at all on SMR.
> F2FS takes a similar approach for SSDs, though it didn't really take
> off because ext4's support is good enough and I suspect that modern
> SSDs are fast enough at erasing.
> 

There is not really a lot to NILFS' structure save the fact it doesn't delete.
It ends up being fairly similar to f2fs. On a SMR w/ TRIM drive this would
imply no (or very little) penalties for write operations as all writes are
actually just appends. I'm not sure the impact it would have on seek/read
time with a normal workload but some people report slight improvements
on SMR drives, especially if they are helium filled, as the denser packing
and lighter gas lead to increased read speeds.

Actually, f2fs may be the better choice, I had almost forgotten about it.
Bigger rollout and more testing. I would need to check feature set in more
detail to make a choice.

---

Weirdly benchmarking tends to show f2fs as deficient to ext4 in most
cases.

For a while I've been interested in f2fs (or now nilfs) as a backing block
layer for a real filesystem. It seems to have a better data model than some
tree based filesystems, but I think we are seeing filesystems suck up
features like snapshotting and logging as a must-have instead of the
older LVM/raid based "storage pipelines."

But then, this is just reimplementing a smart storage controller on your
CPU, though that may be the best place for it.

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Rich Freeman]

On Tue, Jun 23, 2020 at 12:14 PM Sid Spry  wrote:
>
> So if I'm understanding properly most drive firmware won't let you
> operate the device in an append-only mode?

So, there are several types of SMR drives.

There are host-managed, drive-managed, and then hybrid devices that
default to drive-managed for compatibility reasons but the host can
send them a command to take full control so that it is the same as
host-managed.

A host-managed drive just does what the host tells it to.  If the host
tells it to do a write that obliterates some other data on the disk,
the drive just does it, and it is the job of the host
OS/filesystem/application to make sure that they protect any data they
care about.  At the drive level these perform identically to CMR
because they just seek and write like any other drive.  At the
application level these could perform differently since the
application might end up having to work around the drive.  However,
these drives are generally chosen for applications where this is not a
big problem or where the problems can be efficiently mitigated.

A drive-managed drive just looks like a regular drive to the host, and
it ends up having to do a lot of read-before-rewrite operations
because the host is treating it like it is CMR but the drive has to
guarantee that nothing gets lost.  A drive-managed disk has no way to
operate in "append-only" mode.  I'm not an expert in ATA but I believe
disks are just given an LBA and a set of data to write.  Without
support for TRIM the drive has no way to know if it is safe to
overwrite nearby cylinders, which means it has to preserve them.

The biggest problem is that the vendors were trying to conceal the
nature of the drives.  If they advertised TRIM support it would be
pretty obvious they were SMR.

> If any do I suspect
> NILFS (https://en.wikipedia.org/wiki/NILFS) may be a good choice:
>
> "NILFS is a log-structured filesystem, in that the storage medium is treated
> like a circular buffer and new blocks are always written to the end. [...]"
>

On a host-managed disk this would perform the same as on a CMR disk,
with the possible exception of any fixed metadata (I haven't read the
gory details on the filesystem).  If it has no fixed metadata (without
surrounding unused tracks) then it would have no issues at all on SMR.
F2FS takes a similar approach for SSDs, though it didn't really take
off because ext4's support is good enough and I suspect that modern
SSDs are fast enough at erasing.

As I mentioned SSDs have the same issue as SMR, just the scale of the
problem is much smaller.  The erase blocks are much smaller so less
data needs to be read/rewritten following an erase, and then the
operations are all just vastly faster.  Taking 3x longer on a 1us
operation is far different from taking 100x longer on a 100ms
operation.

-- 
Rich

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Sid Spry]

On Tue, Jun 16, 2020, at 7:25 AM, Rich Freeman wrote:
> On Tue, Jun 16, 2020 at 7:36 AM Michael  wrote:
> >
> > Just to add my 2c's before you throw that SMR away, the use case for these
> > drives is to act as disk archives, rather than regular backups.  You write
> > data you want to keep, once.
> 
> If your write pattern is more like a tape SMR should be ok in theory.
> For example, if you wrote to a raw partition using tar (without a
> filesystem) I suspect most SMR implementations (including
> drive-managed) would work tolerably (a host-managed implementation
> would perform identically to CMR).  Once you toss in a filesystem then
> there is no guarantee that the writes will end up being sequential.
> 
> And of course the problem with these latest hidden SMR drives is that
> they generally don't support TRIM, so even repeated sequential writes
> can be a problem because the drive doesn't realize that after you send
> block 1 you're going to send blocks 2-100k all sequentially.  If it
> knew that then it would just start overwriting in place obliterating
> later tracks, since they're just going to be written next anyway.
> Instead this drive is going to cache every write until it can
> consolidate them, which isn't terrible but it still turns every seek
> into three (write buffer, read buffer, write permanent - plus updating
> metadata).  If they weren't being sneaky they could have made it
> drive-managed WITH TRIM so that it worked more like an SSD where you
> get the best performance if the OS uses TRIM, but it can fall back if
> you don't.  Sequential writes on trimmed areas for SMR should perform
> identically to writes on CMR drives.
> 

So if I'm understanding properly most drive firmware won't let you
operate the device in an append-only mode? If any do I suspect
NILFS (https://en.wikipedia.org/wiki/NILFS) may be a good choice:

"NILFS is a log-structured filesystem, in that the storage medium is treated
like a circular buffer and new blocks are always written to the end. [...]"

Realistically I don't know how maintained the filesystem is, and it may
still enforce a hot and warm/cold data separation as a practical concern.
As-is my intended use for these very large drives was not going to be for
hot files anyway; spinning media is too slow. They were going to be running
a snapshottable filesystem and would host my backups.

But if I'm reading it right these drives just suck across the board? I'm
confused. It's like they'd be good at one thing but then they tried to
lie behind the scenes and ended up making the drives good at nothing.

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Sid Spry]

On Tue, Jun 16, 2020, at 2:17 AM, Dale wrote:
> David Haller wrote:
>
>  I mentioned once long ago that I keep a list of frequently used 
> commands. I do that because, well, my memory at times isn't that great. 
> Here is some commands I ran up on based on posts here and what google 
> turned up when searching for things related on those posts. I wanted to 
> share just in case it may help someone else. ;-) dd commands first. 
> 
> 
>  root@fireball / # cat /root/freq-commands | grep dd
>  dd commands
>  dd if=/dev/zero of=/dev/sd bs=4k conv=notrunc
>  dd if=/dev/zero of=/dev/sd bs=4k conv=notrunc oflag=direct #disables cache
>  dd if=/dev/zero of=/dev/sd bs=1M conv=notrunc
>  dd if=<(yes $'\01' | tr -d "\n") of=
>  dd if=<(yes $'\377' | tr -d "\n") of=
>  dd if=<(yes $'\xFF' | tr -d "\n") of= 
>  root@fireball / #
> 
> 
>  The target device or file needs to be added to the end of course on 
> the last three. I tend to leave out some of the target to make sure I 
> don't copy and paste something that ends badly. dd can end badly if 
> targeting the wrong device. 
> 
> 
>  root@fireball / # cat /root/freq-commands | grep smartctl
>  smartctl -t long /dev/sd
>  smartctl -t full ##needs research
>  smartctl -c -t short -d sat /dev/sd ##needs research
>  smartctl -t conveyance -d sat /dev/sd ##needs research
>  smartctl -l selftest -d sat /dev/sd ##needs research
>  smartctl -t  /dev/sd ##needs research
>  smartctl -c /dev/sd ##displays test times in minutes
>  smartctl -l selftest /dev/sd
>  root@fireball / # 
> 

Thanks for these. I do have a general question: has SMART actually shown
anyone predictive capability? In my use and in the use of 4-5 people I know
it only makes you aware of errors after the drive is failing but still online.

I suppose there might be some utility if you're using an error correcting FS,
but for desktop use that was and still is somewhat abnormal.

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

David Haller wrote:
> Hello,
>
> On Mon, 22 Jun 2020, Dale wrote:
>> David Haller wrote:
>>> On Thu, 18 Jun 2020, Dale wrote:
 David Haller wrote:
> [..]
> ./ones | dd of=/dev/null bs=8M count=1000 iflag=fullblock
>>> [..]
 I got it to compile, at least it created a file named ones anyway.  What
 I'm unclear about, where is the if= for dd in the command?  All the
 commands I've seen before has a if= and a of=.  The if for input and of
 for output or target.
>>>  man 1 dd 
>>>if=FILE
>>>   read from FILE instead of stdin
>>> [..]
>>>of=FILE
>>>   write to FILE instead of stdout
>>> 
>>>
>>> note the stuff after 'instead' ;)
>> So it is piped in with the | thingy?  Got it. 
> Exactly. Hm. Maybe 'dev_one' might be a better name for the program...
> Or something ;)
>
> [..]
>> Thanks much.
> You're welcome.
>
> -dnh
>


That name makes sense.  Later on, I might look at that file and say,
what is that for?  Hit delete and down the rabbit hole it goes.  I think
I'll go do that and also edit my notes as well.  Before I forget it.  ;-)

Now I have a way to write ones to a drive or file.  Yeppie! 

Dale

:-)  :-) 

P. S. I hope the new glasses, when they come in, helps my memory.  Never
hurts to hope right? 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[David Haller]

Hello,

On Mon, 22 Jun 2020, Dale wrote:
>David Haller wrote:
>> On Thu, 18 Jun 2020, Dale wrote:
>>> David Haller wrote:
[..]
 ./ones | dd of=/dev/null bs=8M count=1000 iflag=fullblock
>> [..]
>>> I got it to compile, at least it created a file named ones anyway.  What
>>> I'm unclear about, where is the if= for dd in the command?  All the
>>> commands I've seen before has a if= and a of=.  The if for input and of
>>> for output or target.
>>  man 1 dd 
>>if=FILE
>>   read from FILE instead of stdin
>> [..]
>>of=FILE
>>   write to FILE instead of stdout
>> 
>>
>> note the stuff after 'instead' ;)
>
>So it is piped in with the | thingy?  Got it. 

Exactly. Hm. Maybe 'dev_one' might be a better name for the program...
Or something ;)

[..]
>Thanks much.

You're welcome.

-dnh

-- 
/* When we have more time, we can teach the penguin to say 
 * "By your command" or "Activating turbo boost, Michael".
 */
linux-2.2.16/arch/sparc/prom/sun4prom.c

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

David Haller wrote:
> Hello,
>
> On Thu, 18 Jun 2020, Dale wrote:
>> David Haller wrote:
> [..]
>>> Compile with:
>>> gcc $CFLAGS -o ones ones.c
>>> or
>>> gcc $(portageq envvar CFLAGS) -o ones ones.c
>>>
>>> and use/test e.g. like
>>>
>>> ./ones | dd of=/dev/null bs=8M count=1000 iflag=fullblock
> [..]
>> I got it to compile, at least it created a file named ones anyway.  What
>> I'm unclear about, where is the if= for dd in the command?  All the
>> commands I've seen before has a if= and a of=.  The if for input and of
>> for output or target.
>  man 1 dd 
>if=FILE
>   read from FILE instead of stdin
> [..]
>of=FILE
>   write to FILE instead of stdout
> 
>
> note the stuff after 'instead' ;)

So it is piped in with the | thingy?  Got it. 


>> I'm assuming that if I want to target sdb, I'd
>> replace null with /dev/sdb. 
> Yes.
>
>> As I've posted before, even my scripting skills are minimal.  Surprised
>> I got it to compile even.
> Well, you could copy & paste and on gentoo, there is bound to be a
> 'gcc' ;)

Yea, just my skills on that sort of thing is minimal or none.  I do copy
and paste pretty well tho.  ;-) 


>> something.  I placed all this in the /root directory.  I'm assuming I
>> can copy paste the commands above while in /root to make it work?
> Yes. Or put 'ones' in /root/bin/ or even /usr/local/bin. Depending on
> if you expect to use the program later on. But if it's just for this
> once it's fine in /root/ alongside the sourcecode.
>
> HTH,
> -dnh
>


Well, I made notes in my frequent commands file.  It may be rare that I
use it but when I do, maybe the notes will help.  Maybe.  ;-)

Thanks much.

Dale

:-)  :-) 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[David Haller]

Hello,

On Thu, 18 Jun 2020, Dale wrote:
>David Haller wrote:
[..]
>> Compile with:
>> gcc $CFLAGS -o ones ones.c
>> or
>> gcc $(portageq envvar CFLAGS) -o ones ones.c
>>
>> and use/test e.g. like
>>
>> ./ones | dd of=/dev/null bs=8M count=1000 iflag=fullblock
[..]
>I got it to compile, at least it created a file named ones anyway.  What
>I'm unclear about, where is the if= for dd in the command?  All the
>commands I've seen before has a if= and a of=.  The if for input and of
>for output or target.

 man 1 dd 
   if=FILE
  read from FILE instead of stdin
[..]
   of=FILE
  write to FILE instead of stdout


note the stuff after 'instead' ;)

>I'm assuming that if I want to target sdb, I'd
>replace null with /dev/sdb. 

Yes.

>As I've posted before, even my scripting skills are minimal.  Surprised
>I got it to compile even.

Well, you could copy & paste and on gentoo, there is bound to be a
'gcc' ;)

>something.  I placed all this in the /root directory.  I'm assuming I
>can copy paste the commands above while in /root to make it work?

Yes. Or put 'ones' in /root/bin/ or even /usr/local/bin. Depending on
if you expect to use the program later on. But if it's just for this
once it's fine in /root/ alongside the sourcecode.

HTH,
-dnh

-- 
BUGS
   It is not yet possible to change operating system by writ­
   ing to /proc/sys/kernel/ostype. -- Linux sysctl(2) manpage

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

Pengcheng Xu wrote:
> I'm wondering that who's email client is broken, yours or mine?  I'm seeing a 
> lot of U+FFFD replacement characters in this email, like, behind every stop 
> punctuation ("?", ".", and ")"), but not in your other emails on the list.
>
> Regards,

I notice it in your reply too.  I'm not sure.  I use Seamonkey which
seems to autodetect how to format messages.  I've set it to plain text
only for gentoo.org but it sometimes has a mind of its own.  I'm not
sure what else I can do to force it to send plain text.  If anyone using
Seamonkey has a tip on how to do that, I'm all ears.  It's set in
preferences already.

Dale

:-)  :-) 

RE: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Pengcheng Xu]

I'm wondering that who's email client is broken, yours or mine?  I'm seeing a 
lot of U+FFFD replacement characters in this email, like, behind every stop 
punctuation ("?", ".", and ")"), but not in your other emails on the list.

Regards,
-- 
Pengcheng Xu
https://jsteward.moe

> -Original Message-
> From: Dale 
> Sent: Thursday, June 18, 2020 5:14 PM
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] Testing a used hard drive to make SURE it is good.
> 
> David Haller wrote:
> 
> 
>   Hello,
> 
>   On Mon, 15 Jun 2020, Dale wrote:
>   [..]
> 
>   While I'm at it, when running dd, I have zero and random in 
> /dev.�
> Where
>   does a person obtain a one?� In other words, I can write all 
> zeros,
> I
>   can write all random but I can't write all ones since it isn't
> in /dev.�
>   Does that even exist?� Can I create it myself somehow?� Can I
> download
>   it or install it somehow?� I been curious about that for a good
> long
>   while now.� I just never remember to ask.�
> 
> 
>   I've wondered that too. So I just hacked one up just now.
> 
>    ones.c 
>   #include 
>   #include 
>   #include 
>   static unsigned int buf[BUFSIZ];
>   int main(void) {
>   unsigned int i;
>   for(i = 0; i < BUFSIZ; i++) { buf[i] = (unsigned int)-1; }
>   while( write(STDOUT_FILENO, buf, sizeof(buf)) );
>   exit(0);
>   }
>   
> 
>   Compile with:
>   gcc $CFLAGS -o ones ones.c
>   or
>   gcc $(portageq envvar CFLAGS) -o ones ones.c
> 
>   and use/test e.g. like
> 
>   ./ones | dd of=/dev/null bs=8M count=1000 iflag=fullblock
> 
>   Here, it's about as fast as
> 
>   cat /dev/zero | dd of=/dev/null bs=8M count=1000 iflag=fullblock
> 
>   (but only about ~25% as fast as
>   dd if=/dev/zero of=/dev/null bs=8M count=1000 iflag=fullblock
>   for whatever reason ever, but the implementation of /dev/zero is
>   non-trivial ...)
> 
>   HTH,
>   -dnh
> 
> 
> 
> I got it to compile, at least it created a file named ones anyway.� What I'm
> unclear about, where is the if= for dd in the command?� All the commands I've
> seen before has a if= and a of=.� The if for input and of for output or 
> target.�
> I'm assuming that if I want to target sdb, I'd replace null with /dev/sdb.�
> 
> As I've posted before, even my scripting skills are minimal.� Surprised I got
> it to compile even.� lol� Just trying to make sure I don't mess up something.�
> I placed all this in the /root directory.� I'm assuming I can copy paste the
> commands above while in /root to make it work?� I'm asking because I haven't
> tried it yet.�
> 
> Thanks.
> 
> Dale
> 
> :-)� :-)�
> 
> 


openpgp-digital-signature.asc
Description: PGP signature

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

Additional info.  I got the data moved and a smaller drive removed from
the LVM setup.  I decided to play with badblocks a bit and test the old
drive.  I'm testing with destructive patterns since that should
effectively erase data.  The old drive reported a few errors.  I did
some googling and found out that SMART can interfere with badblocks. 
Basically, it accesses the drive and that creates errors.  I found it
odd but decided I'd restart the test and see.  If it had a bad block, it
would still be bad.  I redid the test and so far, it has reported no
errors.  It seems what I found does happen. 

So, if anyone is running badblocks, make sure to turn off SMART for that
device until the tests have completed.  Otherwise you could get negative
results.  This is the command I'm using.

badblocks -wsv -b 4096 /dev/sdb 

If you copy and paste, make sure to edit the device.  When this is all
done, I may post some further details.  Oh, my frequent commands file is
growing.  ;-)

Dale

:-)  :-) 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

David Haller wrote:
> Hello,
>
> On Mon, 15 Jun 2020, Dale wrote:
> [..]
>> While I'm at it, when running dd, I have zero and random in /dev.  Where
>> does a person obtain a one?  In other words, I can write all zeros, I
>> can write all random but I can't write all ones since it isn't in /dev. 
>> Does that even exist?  Can I create it myself somehow?  Can I download
>> it or install it somehow?  I been curious about that for a good long
>> while now.  I just never remember to ask. 
> I've wondered that too. So I just hacked one up just now.
>
>  ones.c 
> #include 
> #include 
> #include 
> static unsigned int buf[BUFSIZ];
> int main(void) {
> unsigned int i;
> for(i = 0; i < BUFSIZ; i++) { buf[i] = (unsigned int)-1; }
> while( write(STDOUT_FILENO, buf, sizeof(buf)) );
> exit(0);
> }
> 
>
> Compile with:
> gcc $CFLAGS -o ones ones.c
> or
> gcc $(portageq envvar CFLAGS) -o ones ones.c
>
> and use/test e.g. like
>
> ./ones | dd of=/dev/null bs=8M count=1000 iflag=fullblock
>
> Here, it's about as fast as
>
> cat /dev/zero | dd of=/dev/null bs=8M count=1000 iflag=fullblock
>
> (but only about ~25% as fast as 
> dd if=/dev/zero of=/dev/null bs=8M count=1000 iflag=fullblock
> for whatever reason ever, but the implementation of /dev/zero is
> non-trivial ...)
>
> HTH,
> -dnh
>

I got it to compile, at least it created a file named ones anyway.  What
I'm unclear about, where is the if= for dd in the command?  All the
commands I've seen before has a if= and a of=.  The if for input and of
for output or target.  I'm assuming that if I want to target sdb, I'd
replace null with /dev/sdb. 

As I've posted before, even my scripting skills are minimal.  Surprised
I got it to compile even.  lol  Just trying to make sure I don't mess up
something.  I placed all this in the /root directory.  I'm assuming I
can copy paste the commands above while in /root to make it work?  I'm
asking because I haven't tried it yet. 

Thanks.

Dale

:-)  :-) 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dr Rainer Woitok]

David,

On Wednesday, 2020-06-17 17:27:30 +0200, you wrote:

> ...
> $ man -l -Tps -P-pa4 $(man -w smartctl ) | ps2pdf - smartctl.pdf

Why not simply

   $ man -Tps -P-pa4 smartctl | ps2pdf - smartctl.pdf

Sincerely,
  Rainer

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[David Haller]

Hello,

On Tue, 16 Jun 2020, Dale wrote:
[..]
>For the /dev/one, I found some which seems to work.  They listed further
>down.  I think my google search terms was poor.  Google doesn't have ESP
>for sure.  O_o
[..]
>dd if=<(yes $'\xFF' | tr -d "\n") of=

This is correct but _much_ slower than my program. I get about
1.1-1.2GB/s with that and with that 'yes' I get only about 47 MB/s...

>root@fireball / # cat /root/freq-commands | grep man
>print man pages to text file
>man  | col -b > /home/dale/Desktop/smartctl.txt
>print man pages to .pdf but has small text.
>man -t  > /home/dale/Desktop/smartctl.pdf

$ man -l -Tps -P-pa4 $(man -w smartctl ) | ps2pdf - smartctl.pdf

Use e.g. '-P-pletter' for letter paper size.

>It's amazing sometimes how wanting to do one thing, leads to learning
>how to do many other things, well, trying to learn how anyway.  LOL 

*g*

-dnh

-- 
I have described my half-hearted attempts at housekeeping as "the
minimual effort required to prevent the fungus from achieving
sentience."  -- Steve VanDevender

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Rich Freeman]

On Wed, Jun 17, 2020 at 8:32 AM Wols Lists  wrote:
>
> On 17/06/20 05:47, Dale wrote:
> > From what I've read, all the drive makers were selling SMR without
> > telling anyone at first.  It wasn't just WD but Seagate as well.
>
> Yes, but Seagate didn't start selling SMR drives advertised as
> "optimised for raid/nas".THAT is what's so bad about the WD case - those
> drives are almost guaranteed to fail the moment anything ELSE goes wrong
> elsewhere.
>

WD also put it in their Black line.  Those are drives marketed to
enthusiasts willing to pay a premium for some kind of cutting-edge
performance.  And they stuck SMR in it.

This is like paying for a high-end NVMe and finding out that it just
has a class 4 SDcard inside with a PCIe-USB2-SD interface.

All the drive manufacturers are scummy at this point though.  You have
no choice but to buy from them, but I trust new drives as far as I can
throw them, and every new drive is getting benchmarked from here on
out...

-- 
Rich

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Wols Lists]

On 17/06/20 05:47, Dale wrote:
> From what I've read, all the drive makers were selling SMR without
> telling anyone at first.  It wasn't just WD but Seagate as well.

Yes, but Seagate didn't start selling SMR drives advertised as
"optimised for raid/nas".THAT is what's so bad about the WD case - those
drives are almost guaranteed to fail the moment anything ELSE goes wrong
elsewhere.

Cheers,
Wol

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Rich Freeman]

On Tue, Jun 16, 2020 at 7:38 PM antlists  wrote:
>
> On 16/06/2020 13:25, Rich Freeman wrote:
> > And of course the problem with these latest hidden SMR drives is that
> > they generally don't support TRIM,
>
> This, I believe, is a problem with the ATA spec. I don't understand
> what's going on, but something like for these drives you need v4 of the
> spec, and only v3 is finalised. Various people have pointed out holes in
> this theory, so you don't need to add to them :-) But yes, I do
> understand that apparently there is no official standard way to send a
> trim to these drives ...

You might be right, but I thought that some drive-managed SMR drives
did support TRIM.  It seems just as likely that TRIM support was not
implemented to conceal the fact that the drives are SMR.

>
> > so even repeated sequential writes
> > can be a problem because the drive doesn't realize that after you send
> > block 1 you're going to send blocks 2-100k all sequentially.  If it
> > knew that then it would just start overwriting in place obliterating
> > later tracks, since they're just going to be written next anyway.
>
> No it can't do that. Because when it overwrites the end of the file it
> will be obliterating other random files that aren't going to be
> overwritten ...

Yes and no.

First, all but the last few tracks can definitely be written without
worry.  The last few tracks might need to be cached but that is far
less seeking than caching the whole thing.

Second, if the writes end on an SMR zone boundary, then nothing will
need rewriting.  This is of course how things are handled on
host-managed SMR drives - you just try to avoid the partial writes in
the first place.

>
> > Instead this drive is going to cache every write until it can
> > consolidate them, which isn't terrible but it still turns every seek
> > into three (write buffer, read buffer, write permanent - plus updating
> > metadata).
>
> Which IS terrible if you don't give the drive down-time to flush the
> buffer ...

It is less terrible than completely random writes, which is my point.
The drive requires far less time to flush the buffer if it is doing
large sequential writes, because only tail ends of a few regions might
need to be preserved.

>
> > If they weren't being sneaky they could have made it
> > drive-managed WITH TRIM so that it worked more like an SSD where you
> > get the best performance if the OS uses TRIM, but it can fall back if
> > you don't.  Sequential writes on trimmed areas for SMR should perform
> > identically to writes on CMR drives.
>
> You're forgetting one thing - rewriting a block on SSD or CMR doesn't
> obliterate neighbouring blocks

Actually, SSDs have exactly the same problem - you cannot rewrite one
block on an SSD without obliterating neighboring blocks.  The
differences are:

1.  The size of these regions is much smaller, so the amount of data
that needs to be read and rewritten is much smaller.  One block of
rewrite might be amplified to hundreds of kilobytes, not hundreds of
megabytes.
2.  SSDs have no seek penalties, so the cost of these extra
reads+writes are much lower in absolute terms, but you're still
getting write amplification.
3.  The SSDs aren't accumulating a large cache over time that can
cause performance to hit a wall for a long time.  The SSD just delays
every write it gets as needed, because these write delays all the time
aren't as noticeable.
4.  SSDs clearly advertise these issues and filesystems were updated
to support TRIM and mitigate them before they were adopted en masse.

#4 is really a big one.  Filesystems that aren't optimized for SSD can
have very noticeable performance drops.  Now, that is all relative - a
non-aware filesystem on SSD is still going to perform way better than
it does on any hard drive, so people switching to SSD from hard drives
are still going to be very happy.  If you are doing benchmarking
though you'll definitely see an issue if you're doing random writes on
SSDs without a filesystem that supports TRIM/etc.

> ... with SMR for every track you rewrite
> you have to salvage the neighbouring track too ...

More than just one neighboring track.  It really is the same issue as
SSD, except:

1.  All the issues are amplified by the seek latency of hard drives.
2.  The overall move from CMR->SMR is 100% downside in performance,
compared to the move from CMR->SSD where the downside is offset by a
much larger upside, so even a dumb implementation feels like a massive
improvement.

In any case, none of this really changes the fact that drive-managed
SMR really shouldn't be a thing at all, and it certainly shouldn't be
an unlabeled "feature."  SMR should have just been introduced as
host-managed as a way to offer cheaper storage for those who have the
use cases and filesystems that can manage it.  There is nothing wrong
with tape for storage, but if you stuck a tape drive preloaded with a
tape in a box that could speak ATA and called it a USB3 external "hard
drive" 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

Dale wrote:
> Howdy,
>
> I finally bought a 8TB drive.  It is used but they claim only a short
> duration.  Still, I want to test it to be sure it is in grade A shape
> before putting a lot of data on it and depending on it.  I am familiar
> with some tools already.  I know about SMART but it is not always
> 100%.  It seems to catch most problems but not all.  I'm familiar with
> dd and writing all zeores or random to it to see if it can in fact
> write to all the parts of the drive but it is slow. It can take a long
> time to write and fill up a 8TB drive. Days maybe??  I googled and
> found a new tool but not sure how accurate it is since I've never used
> it before.  The command is badblocks.  It is installed on my system so
> I'm just curious as to what it will catch that others won't.  Is it
> fast or slow like dd?
>
> I plan to run the SMART test anyway.  It'll take several hours but I'd
> like to run some other test to catch errors that SMART may miss.  If
> there is such a tool that does that.  If you bought a used drive, what
> would you run other than the long version of SMART and its test? 
> Would you spend the time to dd the whole drive?  Would badblocks be a
> better tool?  Is there another better tool for this? 
>
> While I'm at it, when running dd, I have zero and random in /dev. 
> Where does a person obtain a one?  In other words, I can write all
> zeros, I can write all random but I can't write all ones since it
> isn't in /dev.  Does that even exist?  Can I create it myself
> somehow?  Can I download it or install it somehow?  I been curious
> about that for a good long while now.  I just never remember to ask. 
>
> When I add this 8TB drive to /home, I'll have 14TBs of space.  If I
> leave the 3TB drive in instead of swapping it out, I could have about
> 17TBs of space.  O_O 
>
> Thanks to all.
>
> Dale
>
> :-)  :-) 


Update.  I got a lot of info and suggestions from the replies.  Thanks
to all for those.  The drive has passed all the test so far.  I ran a
short and long SMART selftest.  Results:


root@fireball / # smartctl -l selftest /dev/sde
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-5.6.7-gentoo] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status  Remaining 
LifeTime(hours)  LBA_of_first_error
# 1  Short offline   Completed without error   00%
24592 -
# 2  Extended offline    Completed without error   00%
24592 -
# 3  Short offline   Completed without error   00%
24213 -
# 4  Short offline   Completed without error   00%
23493 -
# 5  Short offline   Completed without error   00%
22749 -
# 6  Short offline   Completed without error   00%
22054 -
# 7  Short offline   Completed without error   00%
21310 -
# 8  Short offline   Completed without error   00%
20566 -
# 9  Short offline   Completed without error   00%
19846 -
#10  Short offline   Completed without error   00%
19101 -
#11  Short offline   Completed without error   00%
18381 -
#12  Short offline   Completed without error   00%
17637 -
#13  Short offline   Completed without error   00%
16893 -
#14  Short offline   Completed without error   00%
16173 -
#15  Short offline   Completed without error   00%
12108 -
#16  Short offline   Completed without error   00%
11940 -
#17  Short offline   Completed without error   00%
11772 -
#18  Short offline   Completed without error   00%
11604 -
#19  Short offline   Completed without error   00%
11436 -
#20  Short offline   Completed without error   00%
11268 -
#21  Short offline   Completed without error   00%
11100 -

root@fireball / #


I then ran badblocks to test it.  This is the results of it. 


root@fireball / # badblocks -b 4096 -s -v /dev/sde
Checking blocks 0 to 1953506645
Checking for bad blocks (read-only test):
done
Pass completed, 0 bad blocks found. (0/0/0 errors)
root@fireball / #

It doesn't show it now but I had it show the progress including run
time.  It took about 15 hours to run on this 8TB drive.  If anyone wants
to test a drive in the future, that may help estimate the amount of time
to run this test. 

I tried the conveyance test but this drive doesn't support it.  Since it
shows no errors and it passed the SMART tests as well, I'm thinking it
is time to put data on the thing.  Off to the LVM manual. 

Thanks to all for the tips, tricks and suggestions. 

Dale

:-)  :-) 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

antlists wrote:
> On 16/06/2020 12:26, Dale wrote:
>> I've also read about the resilvering problems too.  I think LVM
>> snapshots and something about BTFS(sp?) has problems.  I've also read
>> that on windoze, it can cause a system to freeze while it is trying
>> to rewrite the moved data too.  It gets so slow, it actually makes
>> the OS not respond.  I suspect it could happen on Linux to if the
>> conditions are right.
>>
> Being all technical, what seems to be happening is ...
>
> Random writes fillup the PMR cache. The drive starts flushing the
> cache, but unfortunately you need a doubly linked list or something -
> you need to be able to find the physical block from the logical
> address (for reading) and to find the logical block from the physical
> address (for cache-flushing). So once the cache fills, the drive needs
> "down time" to move stuff around, and it stops responding to the bus.
> There are reports of disk stalls of 10 minutes or more - bear in mind
> desktop drives are classed as unsuitable for raid because they stall
> for *up* *to* *two* minutes ...
>
>> I guess this is about saving money for the drive makers.  The part
>> that seems to really get under peoples skin tho, them putting those
>> drives out there without telling people that they made changes that
>> affect performance.  It's bad enough for people who use them where
>> they work well but the people that use RAID and such, it seems to
>> bring them to their knees at times.  I can't count the number of
>> times I've read that people support a class action lawsuit over
>> shipping SMR without telling anyone.  It could happen and I'm not
>> sure it shouldn't.  People using RAID and such, especially in some
>> systems, they need performance not drives that beat themselves to death.
>
> Most manufacturers haven't been open, but at least - apart from WD -
> they haven't been stupid either. Bear in mind WD actively market their
> Red drives as suitable for NAS or Raid, putting SMR in there was
> absolutely dumb. Certainly in the UK, as soon as news starts getting
> round, they'll probably find themselves (or rather their retailers
> will get shafted with) loads of returns as "unfit for purpose". And,
> basically, they have a legal liability with no leg to stand on because
> if a product doesn't do what it's advertised for, then the customer is
> *entitled* to a refund.
>
> Dunno why, I've never been a WD fan, so I dodged that bullet. I just
> caught another one, because I regularly advise people they shouldn't
> be running Barracudas, while running two myself ... :-)
>
> Cheers,
> Wol
>
>


>From what I've read, all the drive makers were selling SMR without
telling anyone at first.  It wasn't just WD but Seagate as well.  There
was another maker as well but can't recall what the brand was.  I want
to say HGST but could have been something else.  I tend to like WD and
Seagate and have had a couple Toshibas as well.  I've had a WD go bad
but I've had a Seagate go bad too.  I'm of the mindset that most drives
are good but on occasion, you hit a bad batch.  No matter what brand it
is, there is a horror story out there somewhere.  I've been lucky so
far.  It seems SMART catches that a drive is failing before it actually
does.  I had one that gave the 24 hour warning and it wasn't kidding
either.  Another just starting reporting bad spots.  I replaced it
before it corrupted anything.  I've never lost data that I can recall tho. 

I've read that if there is a lawsuit, the EU will likely be first and
the easiest.  If you say something should work in a certain way and it
doesn't, refund for sure.  Given the large scale of this, lawsuit is
possible.  I'm no lawyer but I do think what the makers did in hiding
this info is wrong.  It doesn't matter what brand it is, they should be
honest about their products.  This is especially true for situations
like RAID, NAS and other 24/7 systems.  Thing is, even my system falls
into that category.  I run 24/7 here except during power failures.  LVM
likely requires a better drive than a regular home type system that is
only used a little each day.  Commercial type systems that are in heavy
use, they require a really heavy duty components.  Claiming something is
or leaving out info that shows they are not is not good.  They should
have known it would bite them at some point.  People have far to many
tools to test drives and uncover the truth. 

Little update.  The drive passed its first SMART long test.  I started
badblocks hours ago and it is almost done.  It's at 96% right now.  I
think it lists bad blocks as it finds them and so far, it hasn't listed
any.  I'll post the results when it is done.  So far, the drive I bought
seems to be in very good condition. 

Now to wait on the last little bit to finish.  Just hope it doesn't get
right to the end and start blowing smoke.  :/

Dale

:-)  :-) 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[antlists]

On 16/06/2020 13:25, Rich Freeman wrote:

And of course the problem with these latest hidden SMR drives is that
they generally don't support TRIM,


This, I believe, is a problem with the ATA spec. I don't understand 
what's going on, but something like for these drives you need v4 of the 
spec, and only v3 is finalised. Various people have pointed out holes in 
this theory, so you don't need to add to them :-) But yes, I do 
understand that apparently there is no official standard way to send a 
trim to these drives ...



so even repeated sequential writes
can be a problem because the drive doesn't realize that after you send
block 1 you're going to send blocks 2-100k all sequentially.  If it
knew that then it would just start overwriting in place obliterating
later tracks, since they're just going to be written next anyway.


No it can't do that. Because when it overwrites the end of the file it 
will be obliterating other random files that aren't going to be 
overwritten ...



Instead this drive is going to cache every write until it can
consolidate them, which isn't terrible but it still turns every seek
into three (write buffer, read buffer, write permanent - plus updating
metadata). 


Which IS terrible if you don't give the drive down-time to flush the 
buffer ...



If they weren't being sneaky they could have made it
drive-managed WITH TRIM so that it worked more like an SSD where you
get the best performance if the OS uses TRIM, but it can fall back if
you don't.  Sequential writes on trimmed areas for SMR should perform
identically to writes on CMR drives.


You're forgetting one thing - rewriting a block on SSD or CMR doesn't 
obliterate neighbouring blocks ... with SMR for every track you rewrite 
you have to salvage the neighbouring track too ...


Cheers,
Wol

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[antlists]

On 16/06/2020 12:26, Dale wrote:
I've also read about the resilvering problems too.  I think LVM 
snapshots and something about BTFS(sp?) has problems.  I've also read 
that on windoze, it can cause a system to freeze while it is trying to 
rewrite the moved data too.  It gets so slow, it actually makes the OS 
not respond.  I suspect it could happen on Linux to if the conditions 
are right.



Being all technical, what seems to be happening is ...

Random writes fillup the PMR cache. The drive starts flushing the cache, 
but unfortunately you need a doubly linked list or something - you need 
to be able to find the physical block from the logical address (for 
reading) and to find the logical block from the physical address (for 
cache-flushing). So once the cache fills, the drive needs "down time" to 
move stuff around, and it stops responding to the bus. There are reports 
of disk stalls of 10 minutes or more - bear in mind desktop drives are 
classed as unsuitable for raid because they stall for *up* *to* *two* 
minutes ...


I guess this is about saving money for the drive makers.  The part that 
seems to really get under peoples skin tho, them putting those drives 
out there without telling people that they made changes that affect 
performance.  It's bad enough for people who use them where they work 
well but the people that use RAID and such, it seems to bring them to 
their knees at times.  I can't count the number of times I've read that 
people support a class action lawsuit over shipping SMR without telling 
anyone.  It could happen and I'm not sure it shouldn't.  People using 
RAID and such, especially in some systems, they need performance not 
drives that beat themselves to death.


Most manufacturers haven't been open, but at least - apart from WD - 
they haven't been stupid either. Bear in mind WD actively market their 
Red drives as suitable for NAS or Raid, putting SMR in there was 
absolutely dumb. Certainly in the UK, as soon as news starts getting 
round, they'll probably find themselves (or rather their retailers will 
get shafted with) loads of returns as "unfit for purpose". And, 
basically, they have a legal liability with no leg to stand on because 
if a product doesn't do what it's advertised for, then the customer is 
*entitled* to a refund.


Dunno why, I've never been a WD fan, so I dodged that bullet. I just 
caught another one, because I regularly advise people they shouldn't be 
running Barracudas, while running two myself ... :-)


Cheers,
Wol

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

Michael wrote:
> On Tuesday, 16 June 2020 12:26:01 BST Dale wrote:
>
>> From what I've read, I agree.  Basically, as some have posted in
>> different places, SMR drives are good when writing once and leaving it
>> alone.  Basically, about like a DVD-R.  From what I've read, let's say I
>> moved a lot of videos around, maybe moved the directory structure
>> around, which means a lot of data to move.  I think I'd risk just
>> putting a new file system on it and then backup everything from
>> scratch.  It may take a little longer given the amount of data but it
>> would be easier on the drive.  It would keep it from hammering as you
>> put it that drive to death. 
>>
>> I've also read about the resilvering problems too.  I think LVM
>> snapshots and something about BTFS(sp?) has problems.  I've also read
>> that on windoze, it can cause a system to freeze while it is trying to
>> rewrite the moved data too.  It gets so slow, it actually makes the OS
>> not respond.  I suspect it could happen on Linux to if the conditions
>> are right.
>>
>> I guess this is about saving money for the drive makers.  The part that
>> seems to really get under peoples skin tho, them putting those drives
>> out there without telling people that they made changes that affect
>> performance.  It's bad enough for people who use them where they work
>> well but the people that use RAID and such, it seems to bring them to
>> their knees at times.  I can't count the number of times I've read that
>> people support a class action lawsuit over shipping SMR without telling
>> anyone.  It could happen and I'm not sure it shouldn't.  People using
>> RAID and such, especially in some systems, they need performance not
>> drives that beat themselves to death.
>>
>> My plan, avoid SMR if at all possible.  Right now, I just don't need the
>> headaches.  The one I got, I'm lucky it works OK, even if it does bump
>> around for quite a while after backups are done. 
>>
>> My new to me hard drive is still testing.  Got a few more hours left
>> yet.  Then I'll run some more tests.  It seems to be OK tho. 
>>
>> Dale
>>
>> :-)  :-) 
> Just to add my 2c's before you throw that SMR away, the use case for these 
> drives is to act as disk archives, rather than regular backups.  You write 
> data you want to keep, once.  SMR disks would work well for your use case of 
> old videos/music/photos you want to keep and won't be overwriting every other 
> day/week/month.  Using rsync with '-c' to compare checksums will also make 
> sure what you've copied is as good/bad as the original fs source.


I try to update about once a day, that way the changes or additions are
fairly small.  On occasion tho, I find a better version of a video which
means I have a new file and delete the old.  That may make it a little
harder for the SMR drive but the amount of data, given my slow DSL, is
not large enough to matter.  I think the biggest changes rsync has
reported so far, about 4 or 5GBs or so. 

My general process is like this.  I find a point where I can backup.  I
power up the external drive, mount it using KDE's Device Notifier, use
rsync to update the files and then unmount the drive with DN.  After
that, I let it sit until I notice that it is not doing that bumping
thing for a bit.  Sometimes that is a couple minutes, sometimes it is 10
or 15 minutes or so.  Generally, it isn't very long really.  Sometimes I
go do something else, cook supper, mow the grass or whatever and cut it
off when I get back. 

In theory I could cut it off right after the backup is done and I've
unmounted it.  Thing is, the changes will build up depending on how
large the cache/buffer/whatever is that it stores as CMR.  From what
I've read, it has a PMR/CMR section and then the rest is SMR.  It writes
new stuff to the PMR/CMR section and when it has time, it moves it to
the SMR parts.  It then does its rewrite thing with the shingles.  I'm
sort of making it simple but that's what some have claimed it does. 

Let's keep in mind, the drive I just bought in this thread is a PMR
drive.  The SMR drive is one I've had a while in a external enclosure. 
Most of the time, it holds my desk down and a stack of Blu-ray discs
up.  That bumpy thing sometimes makes the discs fall off tho.  I need to
clean my desk off, again. 

While I wish my backup drive wasn't a SMR, at least it is acceptable in
performance for what I'm using it for.  If I had spent money on that
drive and put it on /home, then I'd be pretty upset.  We're talking
steam and smoke upset.  It's not like these drives are $20 or $30 or
something.  I got a good deal paying about $150 for this latest new to
me drive.  Still, that's $150 that I don't want to waste on something
that can't handle what I do.  Backup drive that is SMR, well, OK.  I'm
not really pleased about it but it works OK.  Having it on /home where
it could cause my system to freeze or something, well, that may remind
me of the hal days.  I'm sure some recall me and my love 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Rich Freeman]

On Tue, Jun 16, 2020 at 7:36 AM Michael  wrote:
>
> Just to add my 2c's before you throw that SMR away, the use case for these
> drives is to act as disk archives, rather than regular backups.  You write
> data you want to keep, once.

If your write pattern is more like a tape SMR should be ok in theory.
For example, if you wrote to a raw partition using tar (without a
filesystem) I suspect most SMR implementations (including
drive-managed) would work tolerably (a host-managed implementation
would perform identically to CMR).  Once you toss in a filesystem then
there is no guarantee that the writes will end up being sequential.

And of course the problem with these latest hidden SMR drives is that
they generally don't support TRIM, so even repeated sequential writes
can be a problem because the drive doesn't realize that after you send
block 1 you're going to send blocks 2-100k all sequentially.  If it
knew that then it would just start overwriting in place obliterating
later tracks, since they're just going to be written next anyway.
Instead this drive is going to cache every write until it can
consolidate them, which isn't terrible but it still turns every seek
into three (write buffer, read buffer, write permanent - plus updating
metadata).  If they weren't being sneaky they could have made it
drive-managed WITH TRIM so that it worked more like an SSD where you
get the best performance if the OS uses TRIM, but it can fall back if
you don't.  Sequential writes on trimmed areas for SMR should perform
identically to writes on CMR drives.


-- 
Rich

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Michael]

On Tuesday, 16 June 2020 12:26:01 BST Dale wrote:
> Wols Lists wrote:
> > On 16/06/20 10:04, Dale wrote:
> >> I might add, I don't have LVM on that drive.  I read it does not work
> >> well with LVM, RAID etc as you say.  Most likely, that drive will always
> >> be a external drive for backups or something.  If it ever finds itself
> >> on the OS or /home, it'll be a last resort.
> > 
> > LVM it's probably fine with. Raid, MUCH less so. What you need to make
> > sure does NOT happen is a lot of random writes. That might make deleting
> > an lvm snapshot slightly painful ...
> > 
> > But adding a SMR drive to an existing ZFS raid is a guarantee for pain.
> > I don't know why, but "resilvering" causes a lot of random writes. I
> > don't think md-raid behaves this way.
> > 
> > But it's the very nature of raid that, as soon as something goes wrong
> > and a drive needs replacing, everything is going to get hammered. And
> > SMR drives don't take kindly to being hammered ... :-)
> > 
> > Even in normal use, a SMR drive is going to cause grief if it's not
> > handled carefully.
> > 
> > Cheers,
> > Wol
> 
> From what I've read, I agree.  Basically, as some have posted in
> different places, SMR drives are good when writing once and leaving it
> alone.  Basically, about like a DVD-R.  From what I've read, let's say I
> moved a lot of videos around, maybe moved the directory structure
> around, which means a lot of data to move.  I think I'd risk just
> putting a new file system on it and then backup everything from
> scratch.  It may take a little longer given the amount of data but it
> would be easier on the drive.  It would keep it from hammering as you
> put it that drive to death. 
> 
> I've also read about the resilvering problems too.  I think LVM
> snapshots and something about BTFS(sp?) has problems.  I've also read
> that on windoze, it can cause a system to freeze while it is trying to
> rewrite the moved data too.  It gets so slow, it actually makes the OS
> not respond.  I suspect it could happen on Linux to if the conditions
> are right.
> 
> I guess this is about saving money for the drive makers.  The part that
> seems to really get under peoples skin tho, them putting those drives
> out there without telling people that they made changes that affect
> performance.  It's bad enough for people who use them where they work
> well but the people that use RAID and such, it seems to bring them to
> their knees at times.  I can't count the number of times I've read that
> people support a class action lawsuit over shipping SMR without telling
> anyone.  It could happen and I'm not sure it shouldn't.  People using
> RAID and such, especially in some systems, they need performance not
> drives that beat themselves to death.
> 
> My plan, avoid SMR if at all possible.  Right now, I just don't need the
> headaches.  The one I got, I'm lucky it works OK, even if it does bump
> around for quite a while after backups are done. 
> 
> My new to me hard drive is still testing.  Got a few more hours left
> yet.  Then I'll run some more tests.  It seems to be OK tho. 
> 
> Dale
> 
> :-)  :-) 

Just to add my 2c's before you throw that SMR away, the use case for these 
drives is to act as disk archives, rather than regular backups.  You write 
data you want to keep, once.  SMR disks would work well for your use case of 
old videos/music/photos you want to keep and won't be overwriting every other 
day/week/month.  Using rsync with '-c' to compare checksums will also make 
sure what you've copied is as good/bad as the original fs source.

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

Wols Lists wrote:
> On 16/06/20 10:04, Dale wrote:
>> I might add, I don't have LVM on that drive.  I read it does not work
>> well with LVM, RAID etc as you say.  Most likely, that drive will always
>> be a external drive for backups or something.  If it ever finds itself
>> on the OS or /home, it'll be a last resort. 
> LVM it's probably fine with. Raid, MUCH less so. What you need to make
> sure does NOT happen is a lot of random writes. That might make deleting
> an lvm snapshot slightly painful ...
>
> But adding a SMR drive to an existing ZFS raid is a guarantee for pain.
> I don't know why, but "resilvering" causes a lot of random writes. I
> don't think md-raid behaves this way.
>
> But it's the very nature of raid that, as soon as something goes wrong
> and a drive needs replacing, everything is going to get hammered. And
> SMR drives don't take kindly to being hammered ... :-)
>
> Even in normal use, a SMR drive is going to cause grief if it's not
> handled carefully.
>
> Cheers,
> Wol

>From what I've read, I agree.  Basically, as some have posted in
different places, SMR drives are good when writing once and leaving it
alone.  Basically, about like a DVD-R.  From what I've read, let's say I
moved a lot of videos around, maybe moved the directory structure
around, which means a lot of data to move.  I think I'd risk just
putting a new file system on it and then backup everything from
scratch.  It may take a little longer given the amount of data but it
would be easier on the drive.  It would keep it from hammering as you
put it that drive to death. 

I've also read about the resilvering problems too.  I think LVM
snapshots and something about BTFS(sp?) has problems.  I've also read
that on windoze, it can cause a system to freeze while it is trying to
rewrite the moved data too.  It gets so slow, it actually makes the OS
not respond.  I suspect it could happen on Linux to if the conditions
are right.

I guess this is about saving money for the drive makers.  The part that
seems to really get under peoples skin tho, them putting those drives
out there without telling people that they made changes that affect
performance.  It's bad enough for people who use them where they work
well but the people that use RAID and such, it seems to bring them to
their knees at times.  I can't count the number of times I've read that
people support a class action lawsuit over shipping SMR without telling
anyone.  It could happen and I'm not sure it shouldn't.  People using
RAID and such, especially in some systems, they need performance not
drives that beat themselves to death.

My plan, avoid SMR if at all possible.  Right now, I just don't need the
headaches.  The one I got, I'm lucky it works OK, even if it does bump
around for quite a while after backups are done. 

My new to me hard drive is still testing.  Got a few more hours left
yet.  Then I'll run some more tests.  It seems to be OK tho. 

Dale

:-)  :-) 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Wols Lists]

On 16/06/20 10:04, Dale wrote:
> I might add, I don't have LVM on that drive.  I read it does not work
> well with LVM, RAID etc as you say.  Most likely, that drive will always
> be a external drive for backups or something.  If it ever finds itself
> on the OS or /home, it'll be a last resort. 

LVM it's probably fine with. Raid, MUCH less so. What you need to make
sure does NOT happen is a lot of random writes. That might make deleting
an lvm snapshot slightly painful ...

But adding a SMR drive to an existing ZFS raid is a guarantee for pain.
I don't know why, but "resilvering" causes a lot of random writes. I
don't think md-raid behaves this way.

But it's the very nature of raid that, as soon as something goes wrong
and a drive needs replacing, everything is going to get hammered. And
SMR drives don't take kindly to being hammered ... :-)

Even in normal use, a SMR drive is going to cause grief if it's not
handled carefully.

Cheers,
Wol

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

Wols Lists wrote:
> On 16/06/20 08:34, Dale wrote:
>> Right now, I'm backing up to a 8TB external drive, sadly it is a SMR
>> drive but it works.  As I go along, I'll be breaking down my backups. 
>> Example.  I may have my Documents directory, which includes my camera
>> pics, backed up to one drive.  I may have videos backed up to another
>> drive.  Other directories may have to be on other drives.  The biggest
>> things I don't want to lose:  Camera pics that could not be replaced
>> except with a backup.  Videos, some of which are no longer available. 
>> That requires a large drive.  It currently is approaching 6TBs and I
>> have several videos in other locations that are not included in that. 
>> Documents which would be hard to recreate.  Since I have all my emails
>> locally, I don't want to lose those either.  Just a bit ago, I was
>> searching for posts regarding smartctl.  I got quite a few hits.
> Streaming to an SMR should be fine. Doing a cp to a new directory, or
> writing a .tar file, or stuff like that.
>
> What is NOT fine is anything that is likely to result in a lot of
> head-seeking as files and directories get modified ...
>
> Remember that when backing up - so a btrfs with snapshots, or an lvm
> snapshot with rsync in place, is most definitely not a good idea with SMR.
>
> Cheers,
> Wol

Yea, I've read up on them a bit.  They have uses where they work fine
and one can't really tell the difference between it and a PMR/CMR drive.
In my case, it works OK but I have to leave it on a little bit after I
complete my backups and even unmount it.  There was a thread on this
where I asked why I could feel the heads bumping around for a while
after my backup was done.  I think it was Rich that guessed it was a SMR
drive.  Before that, never heard of the thing.  For the small backups I
do every day or two, it works fine.  After some research, it was
discovered that Rich guessed right.  SMR it is. 

I purposely made sure the drive I recently bought was not a SMR drive
tho.  I don't want /home on that, or any other file system for my OS. 
Honestly, I don't plan to buy any SMR drives in the near future.  Maybe
when all the Linux tools figure out how to deal with and manage them. 

I might add, I don't have LVM on that drive.  I read it does not work
well with LVM, RAID etc as you say.  Most likely, that drive will always
be a external drive for backups or something.  If it ever finds itself
on the OS or /home, it'll be a last resort. 

Dale

:-)  :-) 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

Neil Bothwick wrote:
> On Tue, 16 Jun 2020 02:34:55 -0500, Dale wrote:
>
>> Even if I build a NAS setup, I still need a backup arrangement.  Even if
>> I have a RAID setup, still need backups.  It gets complicated for sure. 
>> Sort of expensive too.  Just imagine if my DSL was 10 times faster. 
>> O_O  I'd need to order drives by the case.
> Not necessarily, if the files are going to remain available online, you
> only need to back up the URLs. Downloading again could well be faster
> than restoring from backups. 
>
>


That's the thing, some I have are no longer available anywhere that I
can find.  Even youtube deletes videos with their censorship not because
of copyright but because they don't like the content.  Anytime I find a
good video that I think may be useful later, I download it so that I
have it.  For the most part, it's a good point.  Thing is, my DSL is far
slower than any drive I have, even if it was a USB drive using USB1
speeds.  Downloading is the best way to insure I can watch a video later. 

Of course, videos aren't the only thing I don't want to lose.  I have a
lot of things I don't want to get away from me.  As we know, the more
copies in different locations the safer it is.  I wish I could use a
cloud account but as slow as my download is, upload is even slower, as
usual. 

That said, I'm glad to have the info I have stored here.  I'm just
trying to make sure it doesn't get away from me. 

I wish your way could work tho. 

Dale

:-)  :-) 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Neil Bothwick]

On Tue, 16 Jun 2020 02:34:55 -0500, Dale wrote:

> Even if I build a NAS setup, I still need a backup arrangement.  Even if
> I have a RAID setup, still need backups.  It gets complicated for sure. 
> Sort of expensive too.  Just imagine if my DSL was 10 times faster. 
> O_O  I'd need to order drives by the case.

Not necessarily, if the files are going to remain available online, you
only need to back up the URLs. Downloading again could well be faster
than restoring from backups. 


-- 
Neil Bothwick

Always proofread carefully to see if you any words out.


pgpf0RiIhUgG9.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Wols Lists]

On 16/06/20 08:34, Dale wrote:
> Right now, I'm backing up to a 8TB external drive, sadly it is a SMR
> drive but it works.  As I go along, I'll be breaking down my backups. 
> Example.  I may have my Documents directory, which includes my camera
> pics, backed up to one drive.  I may have videos backed up to another
> drive.  Other directories may have to be on other drives.  The biggest
> things I don't want to lose:  Camera pics that could not be replaced
> except with a backup.  Videos, some of which are no longer available. 
> That requires a large drive.  It currently is approaching 6TBs and I
> have several videos in other locations that are not included in that. 
> Documents which would be hard to recreate.  Since I have all my emails
> locally, I don't want to lose those either.  Just a bit ago, I was
> searching for posts regarding smartctl.  I got quite a few hits.

Streaming to an SMR should be fine. Doing a cp to a new directory, or
writing a .tar file, or stuff like that.

What is NOT fine is anything that is likely to result in a lot of
head-seeking as files and directories get modified ...

Remember that when backing up - so a btrfs with snapshots, or an lvm
snapshot with rsync in place, is most definitely not a good idea with SMR.

Cheers,
Wol

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

William Kenworthy wrote:
>
> In case no one has mentioned it, check out "stress" and "stress-ng" -
> they have HDD tests available. (I am going to have to look into that
> --ignite-cpu option ... :)
>
> BillK
>
>

I did see that mentioned somewhere but forgot about it.  Another
option.  May have to edit the frequent commands file again. 

Thanks.

Dale

:-)  :-) 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

Mark Knecht wrote:
>
>
> On Mon, Jun 15, 2020 at 12:37 PM Dale  > wrote:
> >
> > Howdy,
> >
> > I finally bought a 8TB drive.  It is used but they claim only a
> short duration.  Still, I want to test it to be sure it is in grade A
> shape before putting a lot of data on it and depending on it.  I am
> familiar with some tools already.  I know about SMART but it is not
> always 100%.  It seems to catch most problems but not all.  I'm
> familiar with dd and writing all zeores or random to it to see if it
> can in fact write to all the parts of the drive but it is slow. It can
> take a long time to write and fill up a 8TB drive. Days maybe??  I
> googled and found a new tool but not sure how accurate it is since
> I've never used it before.  The command is badblocks.  It is installed
> on my system so I'm just curious as to what it will catch that others
> won't.  Is it fast or slow like dd?
> >
> > I plan to run the SMART test anyway.  It'll take several hours but
> I'd like to run some other test to catch errors that SMART may miss. 
> If there is such a tool that does that.  If you bought a used drive,
> what would you run other than the long version of SMART and its test? 
> Would you spend the time to dd the whole drive?  Would badblocks be a
> better tool?  Is there another better tool for this?
> >
> > While I'm at it, when running dd, I have zero and random in /dev. 
> Where does a person obtain a one?  In other words, I can write all
> zeros, I can write all random but I can't write all ones since it
> isn't in /dev.  Does that even exist?  Can I create it myself
> somehow?  Can I download it or install it somehow?  I been curious
> about that for a good long while now.  I just never remember to ask.
> >
> > When I add this 8TB drive to /home, I'll have 14TBs of space.  If I
> leave the 3TB drive in instead of swapping it out, I could have about
> 17TBs of space.  O_O
> >
> > Thanks to all.
> >
> > Dale
> >
> > :-)  :-)
>
> The SMART test, long version, will do a very reasonable job catching
> problems. Run it 2 or 3 times if it makes you feel better.
>
> Chris's suggestion about Spinrite is another option but it is slow,
> slow, slow. Might take you weeks? On a drive that large if it worked
> at all.
>
> As an aside, but important, I fear that you're possibly falling into
> the trap most of us do at home. Please don't. Once you have 17TB of
> space on your system how are you planning on doing your weekly
> backups? Do you have 17TB+ on an external drive or system? Will you
> back up to BlueRay discs or something like that?
>
> Mark


Way back, we used Spinrite to test drives.  Think mid 90's.  Yea, it was
slow then on what today is a tiny hard drive.  Can't imagine modern
drive sizes.  It is good tho.  It reads/writes every single part of a
drive.  It will generally find fault if there is one. 

Right now, I'm backing up to a 8TB external drive, sadly it is a SMR
drive but it works.  As I go along, I'll be breaking down my backups. 
Example.  I may have my Documents directory, which includes my camera
pics, backed up to one drive.  I may have videos backed up to another
drive.  Other directories may have to be on other drives.  The biggest
things I don't want to lose:  Camera pics that could not be replaced
except with a backup.  Videos, some of which are no longer available. 
That requires a large drive.  It currently is approaching 6TBs and I
have several videos in other locations that are not included in that. 
Documents which would be hard to recreate.  Since I have all my emails
locally, I don't want to lose those either.  Just a bit ago, I was
searching for posts regarding smartctl.  I got quite a few hits.

Even if I build a NAS setup, I still need a backup arrangement.  Even if
I have a RAID setup, still need backups.  It gets complicated for sure. 
Sort of expensive too.  Just imagine if my DSL was 10 times faster. 
O_O  I'd need to order drives by the case.

Dale

:-)  :-) 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[William Kenworthy]

In case no one has mentioned it, check out "stress" and "stress-ng" -
they have HDD tests available. (I am going to have to look into that
--ignite-cpu option ... :)

BillK

On 16/6/20 3:17 pm, Dale wrote:
> David Haller wrote:
>> Hello,
>>
>> On Mon, 15 Jun 2020, Dale wrote:
>> [..]
>>> While I'm at it, when running dd, I have zero and random in /dev.  Where
>>> does a person obtain a one?  In other words, I can write all zeros, I
>>> can write all random but I can't write all ones since it isn't in /dev. 
>>> Does that even exist?  Can I create it myself somehow?  Can I download
>>> it or install it somehow?  I been curious about that for a good long
>>> while now.  I just never remember to ask. 
>> I've wondered that too. So I just hacked one up just now.
>>
>>  ones.c 
>> #include 
>> #include 
>> #include 
>> static unsigned int buf[BUFSIZ];
>> int main(void) {
>> unsigned int i;
>> for(i = 0; i < BUFSIZ; i++) { buf[i] = (unsigned int)-1; }
>> while( write(STDOUT_FILENO, buf, sizeof(buf)) );
>> exit(0);
>> }
>> 
>>
>> Compile with:
>> gcc $CFLAGS -o ones ones.c
>> or
>> gcc $(portageq envvar CFLAGS) -o ones ones.c
>>
>> and use/test e.g. like
>>
>> ./ones | dd of=/dev/null bs=8M count=1000 iflag=fullblock
>>
>> Here, it's about as fast as
>>
>> cat /dev/zero | dd of=/dev/null bs=8M count=1000 iflag=fullblock
>>
>> (but only about ~25% as fast as 
>> dd if=/dev/zero of=/dev/null bs=8M count=1000 iflag=fullblock
>> for whatever reason ever, but the implementation of /dev/zero is
>> non-trivial ...)
>>
>> HTH,
>> -dnh
>>
>
>
> Thanks David for the reply and others as well.  I got some good ideas
> from some experts plus gave me things to google.  More further down.
>
> For the /dev/one, I found some which seems to work.  They listed
> further down.  I think my google search terms was poor.  Google
> doesn't have ESP for sure.  O_o
>
> I mentioned once long ago that I keep a list of frequently used
> commands.  I do that because, well, my memory at times isn't that
> great.  Here is some commands I ran up on based on posts here and what
> google turned up when searching for things related on those posts.  I
> wanted to share just in case it may help someone else.  ;-)  dd
> commands first. 
>
>
> root@fireball / # cat /root/freq-commands | grep dd
> dd commands
> dd if=/dev/zero of=/dev/sd bs=4k conv=notrunc
> dd if=/dev/zero of=/dev/sd bs=4k conv=notrunc oflag=direct  #disables
> cache
> dd if=/dev/zero of=/dev/sd bs=1M conv=notrunc
> dd if=<(yes $'\01' | tr -d "\n") of=
> dd if=<(yes $'\377' | tr -d "\n") of=
> dd if=<(yes $'\xFF' | tr -d "\n") of=
> root@fireball / #
>
>
> The target device or file needs to be added to the end of course on
> the last three.  I tend to leave out some of the target to make sure I
> don't copy and paste something that ends badly.  dd can end badly if
> targeting the wrong device. 
>
>
> root@fireball / # cat /root/freq-commands | grep smartctl
> smartctl -t long /dev/sd
> smartctl -t full  ##needs research
> smartctl -c -t short -d sat /dev/sd  ##needs research
> smartctl -t conveyance -d sat /dev/sd  ##needs research
> smartctl -l selftest -d sat /dev/sd  ##needs research
> smartctl -t  /dev/sd  ##needs research
> smartctl -c /dev/sd  ##displays test times in minutes
> smartctl -l selftest /dev/sd
> root@fireball / #
>
>
> The ones where I have 'needs research' on the end, I'm still checking
> the syntax of the command.  I haven't quite found exact examples of
> them yet.  This also led to me wanting to print the man page for
> smartctl.  That is a task in itself.  Still, google found me some
> options which are here:
>
>
> root@fireball / # cat /root/freq-commands | grep man
> print man pages to text file
> man  | col -b > /home/dale/Desktop/smartctl.txt
> print man pages to .pdf but has small text.
> man -t  > /home/dale/Desktop/smartctl.pdf
> root@fireball / #
>
>
> It's amazing sometimes how wanting to do one thing, leads to learning
> how to do many other things, well, trying to learn how anyway.  LOL 
>
> I started the smartctl longtest a while ago.  It's still running but
> it hasn't let the smoke out yet.  It's a good sign I guess. I only
> have one SATA port left now.  I got to order another PCI SATA card I
> guess.  :/  I really need to think on the NAS project. 
>
> Thanks to all. 
>
> Dale
>
> :-)  :-) 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

David Haller wrote:
> Hello,
>
> On Mon, 15 Jun 2020, Dale wrote:
> [..]
>> While I'm at it, when running dd, I have zero and random in /dev.  Where
>> does a person obtain a one?  In other words, I can write all zeros, I
>> can write all random but I can't write all ones since it isn't in /dev. 
>> Does that even exist?  Can I create it myself somehow?  Can I download
>> it or install it somehow?  I been curious about that for a good long
>> while now.  I just never remember to ask. 
> I've wondered that too. So I just hacked one up just now.
>
>  ones.c 
> #include 
> #include 
> #include 
> static unsigned int buf[BUFSIZ];
> int main(void) {
> unsigned int i;
> for(i = 0; i < BUFSIZ; i++) { buf[i] = (unsigned int)-1; }
> while( write(STDOUT_FILENO, buf, sizeof(buf)) );
> exit(0);
> }
> 
>
> Compile with:
> gcc $CFLAGS -o ones ones.c
> or
> gcc $(portageq envvar CFLAGS) -o ones ones.c
>
> and use/test e.g. like
>
> ./ones | dd of=/dev/null bs=8M count=1000 iflag=fullblock
>
> Here, it's about as fast as
>
> cat /dev/zero | dd of=/dev/null bs=8M count=1000 iflag=fullblock
>
> (but only about ~25% as fast as 
> dd if=/dev/zero of=/dev/null bs=8M count=1000 iflag=fullblock
> for whatever reason ever, but the implementation of /dev/zero is
> non-trivial ...)
>
> HTH,
> -dnh
>


Thanks David for the reply and others as well.  I got some good ideas
from some experts plus gave me things to google.  More further down.

For the /dev/one, I found some which seems to work.  They listed further
down.  I think my google search terms was poor.  Google doesn't have ESP
for sure.  O_o

I mentioned once long ago that I keep a list of frequently used
commands.  I do that because, well, my memory at times isn't that
great.  Here is some commands I ran up on based on posts here and what
google turned up when searching for things related on those posts.  I
wanted to share just in case it may help someone else.  ;-)  dd commands
first. 


root@fireball / # cat /root/freq-commands | grep dd
dd commands
dd if=/dev/zero of=/dev/sd bs=4k conv=notrunc
dd if=/dev/zero of=/dev/sd bs=4k conv=notrunc oflag=direct  #disables cache
dd if=/dev/zero of=/dev/sd bs=1M conv=notrunc
dd if=<(yes $'\01' | tr -d "\n") of=
dd if=<(yes $'\377' | tr -d "\n") of=
dd if=<(yes $'\xFF' | tr -d "\n") of=
root@fireball / #


The target device or file needs to be added to the end of course on the
last three.  I tend to leave out some of the target to make sure I don't
copy and paste something that ends badly.  dd can end badly if targeting
the wrong device. 


root@fireball / # cat /root/freq-commands | grep smartctl
smartctl -t long /dev/sd
smartctl -t full  ##needs research
smartctl -c -t short -d sat /dev/sd  ##needs research
smartctl -t conveyance -d sat /dev/sd  ##needs research
smartctl -l selftest -d sat /dev/sd  ##needs research
smartctl -t  /dev/sd  ##needs research
smartctl -c /dev/sd  ##displays test times in minutes
smartctl -l selftest /dev/sd
root@fireball / #


The ones where I have 'needs research' on the end, I'm still checking
the syntax of the command.  I haven't quite found exact examples of them
yet.  This also led to me wanting to print the man page for smartctl. 
That is a task in itself.  Still, google found me some options which are
here:


root@fireball / # cat /root/freq-commands | grep man
print man pages to text file
man  | col -b > /home/dale/Desktop/smartctl.txt
print man pages to .pdf but has small text.
man -t  > /home/dale/Desktop/smartctl.pdf
root@fireball / #


It's amazing sometimes how wanting to do one thing, leads to learning
how to do many other things, well, trying to learn how anyway.  LOL 

I started the smartctl longtest a while ago.  It's still running but it
hasn't let the smoke out yet.  It's a good sign I guess. I only have one
SATA port left now.  I got to order another PCI SATA card I guess.  :/ 
I really need to think on the NAS project. 

Thanks to all. 

Dale

:-)  :-) 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[madscientistatlarge]

I just tested a couple of 4TB drives, the hgst 7k400 series (7200 rpm), it 
takes about a day to fill one of those, copying from another one of those. I 
prefer to use "real" data, rather than a test pattern. My procedure is to first 
run the short SMART test, format the drive (which also takes awhile..) and fill 
it with data. I like to copy to the test drive from 2 drives at the same time, 
producing fragmentation, and then copy what's on the test drive already into a 
new folder on the test drive and repeat, it makes it a brutal mechanical test. 
After that I fsck it, power cycle the computer/drive, run fsck again, and then 
run the drives long SMART test, which does take a looong time, and then run 
badblocks which should find nothing. I've had a drive that tested fine but 
would corrupt data (I don't know if the drive electronics started doing random 
writes or what, this wasn't an '80's drive), often within minutes, which is 
part of why I power cycle, and why I use fsck after putting data on the drive. 
I of course I then reformat the drive. It does bloody take days, plan to not be 
watching it. I've done this on my main machine as it doesn't hurt anything 
else. These are Refurbished drives in this case, going into a raid 6 to try and 
end bitrot, which I"ve seen too much of using similiar drives, it tends to 
slowly delete scenes from video when this happens without any indication (other 
than not showing things you know are in the video from watching before).

It's a major pain, but so is watching stuff get corrupted! Note that a failing 
power supply can make a drive look like it's failed, seen this a couple of 
months ago. As soon as I put the drive in question on an external supply it 
magically worked fine. Did this right after another drive appeared to fail the 
same way (different brand). Apparently one drive was more sensitive to bad 
power then the other, both drives are in fact fine! I've also seen ram 
repeatedly test bad with a failing power supply. Not really surprising that a 
failing power supply can make things look bad. What was unexpected was the 
subtlety of the failure over time. Lesson being to always test drives that just 
failed on another machine, or with external supply and usb adapter after the 
machine boots to an OS (also great drives that have failed so that they 
delay/block even the BIOS from coming up, which failing drives can do). 
Obviously checking the power supply is always a good idea, though substituting 
another supply works. Power supply failures are not always apparent with a 
voltmeter as they can be noisy and your' meter won't tell you that generally. 
Also of course, ALL of this can be temperature sensitive and hence 
intermittent, I've done repair work, intermittent failures can take you all 
over trying to find them, random correlation abounds and misleads! So if the 
drive does appear to fail, it may or may not be the drive. I mention this as 
it's really annoying to find the real issue after replacing parts that weren't 
bad and possibly throwing out perfectly good hardware.

I did get a refurbished drive that was invisible to the bios, and wouldn't work 
externally on a different supply either. Can't be sure until you try it on a 
different machine or external.

Sent with [ProtonMail](https://protonmail.com) Secure Email.

‐‐‐ Original Message ‐‐‐
On Monday, June 15, 2020 10:07 AM, Dale  wrote:

> Howdy,
>
> I finally bought a 8TB drive. It is used but they claim only a short 
> duration. Still, I want to test it to be sure it is in grade A shape before 
> putting a lot of data on it and depending on it. I am familiar with some 
> tools already. I know about SMART but it is not always 100%. It seems to 
> catch most problems but not all. I'm familiar with dd and writing all zeores 
> or random to it to see if it can in fact write to all the parts of the drive 
> but it is slow. It can take a long time to write and fill up a 8TB drive. 
> Days maybe?? I googled and found a new tool but not sure how accurate it is 
> since I've never used it before. The command is badblocks. It is installed on 
> my system so I'm just curious as to what it will catch that others won't. Is 
> it fast or slow like dd?
>
> I plan to run the SMART test anyway. It'll take several hours but I'd like to 
> run some other test to catch errors that SMART may miss. If there is such a 
> tool that does that. If you bought a used drive, what would you run other 
> than the long version of SMART and its test? Would you spend the time to dd 
> the whole drive? Would badblocks be a better tool? Is there another better 
> tool for this?
>
> While I'm at it, when running dd, I have zero and random in /dev. Where does 
> a person obtain a one? In other words, I can write all zeros, I can write all 
> random but I can't write all ones since it isn't in /dev. Does that even 
> exist? Can I create it myself somehow? Can I download it or install it 
> somehow? I been curious about 

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[David Haller]

Hello,

On Mon, 15 Jun 2020, Dale wrote:
[..]
>While I'm at it, when running dd, I have zero and random in /dev.  Where
>does a person obtain a one?  In other words, I can write all zeros, I
>can write all random but I can't write all ones since it isn't in /dev. 
>Does that even exist?  Can I create it myself somehow?  Can I download
>it or install it somehow?  I been curious about that for a good long
>while now.  I just never remember to ask. 

I've wondered that too. So I just hacked one up just now.

 ones.c 
#include 
#include 
#include 
static unsigned int buf[BUFSIZ];
int main(void) {
unsigned int i;
for(i = 0; i < BUFSIZ; i++) { buf[i] = (unsigned int)-1; }
while( write(STDOUT_FILENO, buf, sizeof(buf)) );
exit(0);
}


Compile with:
gcc $CFLAGS -o ones ones.c
or
gcc $(portageq envvar CFLAGS) -o ones ones.c

and use/test e.g. like

./ones | dd of=/dev/null bs=8M count=1000 iflag=fullblock

Here, it's about as fast as

cat /dev/zero | dd of=/dev/null bs=8M count=1000 iflag=fullblock

(but only about ~25% as fast as 
dd if=/dev/zero of=/dev/null bs=8M count=1000 iflag=fullblock
for whatever reason ever, but the implementation of /dev/zero is
non-trivial ...)

HTH,
-dnh

-- 
Computers make very fast, very accurate mistakes.

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Mark Knecht]

On Mon, Jun 15, 2020 at 1:01 PM Rich Freeman  wrote:
>
> On Mon, Jun 15, 2020 at 3:54 PM Mark Knecht  wrote:
> >
> > The SMART test, long version, will do a very reasonable job catching
problems. Run it 2 or 3 times if it makes you feel better.
> >
> > Chris's suggestion about Spinrite is another option but it is slow,
slow, slow. Might take you weeks? On a drive that large if it worked at all.
>
> Is spinrite really still of any use in this era?  I'm skeptical of
> that.  Maybe in the pre-IDE days it made more sense.
>

I used it a few years ago on some 2TB or 3TB drives, more out of curiosity
when I found my old license. Other than that I agree with you Rich

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Rich Freeman]

On Mon, Jun 15, 2020 at 3:54 PM Mark Knecht  wrote:
>
> The SMART test, long version, will do a very reasonable job catching 
> problems. Run it 2 or 3 times if it makes you feel better.
>
> Chris's suggestion about Spinrite is another option but it is slow, slow, 
> slow. Might take you weeks? On a drive that large if it worked at all.

Is spinrite really still of any use in this era?  I'm skeptical of
that.  Maybe in the pre-IDE days it made more sense.

I'd echo the SMART test.  If you don't want to trust the vendor then I run:
badblocks -b 4096 -v -w -s /dev/disk/by-id/...

Note that this is a DESTRUCTIVE write test.  There are also
non-destructive read-only tests but obviously this is more limited in
capability.  It will write a series of patterns to the disk and read
them back.  Barring some really clever firmware it should detect any
kind of media error.  I think there might be software that writes
pseudo-random patterns and then reads them back, which would defeat
any firmware fiddling.

For a drive in the 10-12TB size you're talking about a day or so per
pass, so 4 passes will take a few days.  Can't get around 12TB /
(120MB/s).  If I'm not in a hurry to use the space I'll still do it as
nothing is worse than starting to migrate onto a drive and then having
to back out, and that is with redundant data.

-- 
Rich

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Mark Knecht]

On Mon, Jun 15, 2020 at 12:37 PM Dale  wrote:
>
> Howdy,
>
> I finally bought a 8TB drive.  It is used but they claim only a short
duration.  Still, I want to test it to be sure it is in grade A shape
before putting a lot of data on it and depending on it.  I am familiar with
some tools already.  I know about SMART but it is not always 100%.  It
seems to catch most problems but not all.  I'm familiar with dd and writing
all zeores or random to it to see if it can in fact write to all the parts
of the drive but it is slow. It can take a long time to write and fill up a
8TB drive. Days maybe??  I googled and found a new tool but not sure how
accurate it is since I've never used it before.  The command is badblocks.
It is installed on my system so I'm just curious as to what it will catch
that others won't.  Is it fast or slow like dd?
>
> I plan to run the SMART test anyway.  It'll take several hours but I'd
like to run some other test to catch errors that SMART may miss.  If there
is such a tool that does that.  If you bought a used drive, what would you
run other than the long version of SMART and its test?  Would you spend the
time to dd the whole drive?  Would badblocks be a better tool?  Is there
another better tool for this?
>
> While I'm at it, when running dd, I have zero and random in /dev.  Where
does a person obtain a one?  In other words, I can write all zeros, I can
write all random but I can't write all ones since it isn't in /dev.  Does
that even exist?  Can I create it myself somehow?  Can I download it or
install it somehow?  I been curious about that for a good long while now.
I just never remember to ask.
>
> When I add this 8TB drive to /home, I'll have 14TBs of space.  If I leave
the 3TB drive in instead of swapping it out, I could have about 17TBs of
space.  O_O
>
> Thanks to all.
>
> Dale
>
> :-)  :-)

The SMART test, long version, will do a very reasonable job catching
problems. Run it 2 or 3 times if it makes you feel better.

Chris's suggestion about Spinrite is another option but it is slow, slow,
slow. Might take you weeks? On a drive that large if it worked at all.

As an aside, but important, I fear that you're possibly falling into the
trap most of us do at home. Please don't. Once you have 17TB of space on
your system how are you planning on doing your weekly backups? Do you have
17TB+ on an external drive or system? Will you back up to BlueRay discs or
something like that?

Mark

Re: [gentoo-user] Testing a used hard drive to make SURE it is good.

[Spackman, Chris]

On 2020/06/15 at 11:07am, Dale wrote:

> I finally bought a 8TB drive.  It is used but they claim only a short
> duration.  Still, I want to test it to be sure it is in grade A shape
> before putting a lot of data on it and depending on it.  I am familiar
> with some tools already.  I know about SMART but it is not always 100%. 
> It seems to catch most problems but not all.  I'm familiar with dd and
> writing all zeores or random to it to see if it can in fact write to all
> the parts of the drive but it is slow. It can take a long time to write
> and fill up a 8TB drive. Days maybe??  I googled and found a new tool
> but not sure how accurate it is since I've never used it before.  The
> command is badblocks.  It is installed on my system so I'm just curious
> as to what it will catch that others won't.  Is it fast or slow like dd?

If you have a few days, I'd run spinrite
(https://www.grc.com/sr/spinrite.htm) on it. It is very slow but very
good at checking and repairing disks. (Gibson is working on an updated
version that will be much, much faster.) No joke that you would have to
leave the current version running for several days (at least) for an 8TB
drive, if you ran the most comprehensive test.

Aside from time, the drawback is that spinrite costs money.

I've used badblocks and it is not fast, but not nearly as slow as
spinrite. IIRC, it took maybe a few hours for a 250GB drive. That said,
I've been lucky not to have a huge number of problems with drives, so I
can't say how much either has helped.

-- 
Chris Spackman  ch...@osugisakae.com

ESL Coordinator The Graham Family of Schools
ESL Instructor  Columbus State Community College
Japan Exchange and Teaching Program   Wajima, Ishikawa 1995-1998
Linux user since 1998 Linux User #137532

[gentoo-user] Testing a used hard drive to make SURE it is good.

[Dale]

Howdy,

I finally bought a 8TB drive.  It is used but they claim only a short
duration.  Still, I want to test it to be sure it is in grade A shape
before putting a lot of data on it and depending on it.  I am familiar
with some tools already.  I know about SMART but it is not always 100%. 
It seems to catch most problems but not all.  I'm familiar with dd and
writing all zeores or random to it to see if it can in fact write to all
the parts of the drive but it is slow. It can take a long time to write
and fill up a 8TB drive. Days maybe??  I googled and found a new tool
but not sure how accurate it is since I've never used it before.  The
command is badblocks.  It is installed on my system so I'm just curious
as to what it will catch that others won't.  Is it fast or slow like dd?

I plan to run the SMART test anyway.  It'll take several hours but I'd
like to run some other test to catch errors that SMART may miss.  If
there is such a tool that does that.  If you bought a used drive, what
would you run other than the long version of SMART and its test?  Would
you spend the time to dd the whole drive?  Would badblocks be a better
tool?  Is there another better tool for this? 

While I'm at it, when running dd, I have zero and random in /dev.  Where
does a person obtain a one?  In other words, I can write all zeros, I
can write all random but I can't write all ones since it isn't in /dev. 
Does that even exist?  Can I create it myself somehow?  Can I download
it or install it somehow?  I been curious about that for a good long
while now.  I just never remember to ask. 

When I add this 8TB drive to /home, I'll have 14TBs of space.  If I
leave the 3TB drive in instead of swapping it out, I could have about
17TBs of space.  O_O 

Thanks to all.

Dale

:-)  :-)