Re: [gentoo-user] External hard drive and idle activity

[Dale]

Wols Lists wrote:
> On 04/08/20 08:42, Wols Lists wrote:
>> Both LVM and btrfs offer snapshotting, so you take a snapshot before
>> doing an in-place rsync, giving you one full backup per snapshot, but
>> the drive is actually only storing the changes between snapshots.
>> Probably run the backup much faster too.
> Just strikes me this would be near ideal for an SMR drive, because this
> would be copy-on-write, so the backup would just be streaming new data
> to disk.
>
> And by judiciously choosing when to delete snapshots, you have
> considerable control over when the drive decides to do a defrag.
>
> Cheers,
> Wol
>
>


I've never used those features of LVM before.  Most likely, I should. 
I've had occasion to do a backup and then wish I had a old file back
that was deleted during the new backup process.  Example.  I have a copy
of a video for a particular show.  I find a better version, HD or
something, and download it.  I then remove the old one and find out
shortly after that it's the wrong episode or something.  At that point,
I'm missing a episode.  I'd rather have a standard definition version
than none at all.  I suspect doing it the way you mention I'd be able to
get that old copy back provided that snapshot hasn't been deleted yet. 
The way I do it now, once I update the backups, old stuff is deleted. 

Also, I just did another fairly large update on the backups.  Once it
hit around 50GBs or so, it started slowing down again.  It was even
slower than last time.  It was transferring at around 70MBs/sec.  Of
course, it could be partly because that drive is filling up.  It's
around 80% or so. 

Anyway.  I need to look into the snapshot thing.  Gotta find a howto.  ;-)

Dale

:-)  :-) 

Re: [gentoo-user] External hard drive and idle activity

[Wols Lists]

On 04/08/20 08:42, Wols Lists wrote:
> Both LVM and btrfs offer snapshotting, so you take a snapshot before
> doing an in-place rsync, giving you one full backup per snapshot, but
> the drive is actually only storing the changes between snapshots.
> Probably run the backup much faster too.

Just strikes me this would be near ideal for an SMR drive, because this
would be copy-on-write, so the backup would just be streaming new data
to disk.

And by judiciously choosing when to delete snapshots, you have
considerable control over when the drive decides to do a defrag.

Cheers,
Wol

Re: [gentoo-user] External hard drive and idle activity

[Wols Lists]

On 04/08/20 04:17, Dale wrote:
> This drive is formatted with ext4.  It doesn't have LVM or anything just
> straight ext4.  Given it is external, I didn't see the point of having
> LVM on it and adding another layer to deal with when there is no
> benefits to it. 

LVM has a big advantage if not much (relatively) changes between
backups. If you have let's say 900GB of data, your backup disk is 1TB,
and say 20GB of data changes between backups, with LVM that drive could
store 5 *full* backups! You could use btrfs to the same effect.

Both LVM and btrfs offer snapshotting, so you take a snapshot before
doing an in-place rsync, giving you one full backup per snapshot, but
the drive is actually only storing the changes between snapshots.
Probably run the backup much faster too.

If this drive is used to store full backups, and only stores the one
copy, it won't be that expensive/risky to reformat and try that?

Cheers,
Wol

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Rich Freeman wrote:
> On Mon, Aug 3, 2020 at 8:24 AM Dale  wrote:
>> In the past, I've never seen the drive on the larger files be that slow even 
>> toward the end.  Generally, it stays pretty close to 180MBs/sec or so which 
>> is what I usually get with PMR drives.
> Yeah, just hard to be certain without ditching the filesystem layer or
> doing some kind of comparison.  The difference in write speed across
> the drive on recent drives I've gotten is more pronounced than I've
> seen on other drives, but these drives tend to be around 12TB.
>
> Definitely the thing to watch out for is a big drop in transfer rate
> once a large number of blocks have been transferred continuously, and
> then performance returns after you let the drive thrash for a while.
> I've seen complaints of zfs rebuilds going from hours/days to
> weeks/months in length, so it isn't just a 50% drop when you're doing
> worst-case access patterns.  On the other hand I hear that mdadm isn't
> so bad, so if the writes are sequential the drive might be better at
> skipping the cache, and maybe zfs just does its rebuild
> non-sequentially (which isn't really ideal anyway).
>
> I haven't really dug into the guts of how zfs metadata works, but with
> btrfs I believe the chunks are basically their own layer, and the
> filesystem can scrub them without really any care about what files are
> stored in them.  That allows them to be easily scrubbed sequentially.
> When I did rebuilds on btrfs they tended to run at about the max
> throughput of the drives as long as there wasn't any other access
> going on.  It can also do read-only scrubs to check data integrity
> sequentially across the disk, which suggests the checksums are stored
> at a lower layer and so the data can be verified without worrying
> about file fragmentation and so on.  This layering also lets btrfs
> switch "RAID modes" on the fly with half of the disk being RAID1 and
> half the disk being RAID5 and so on - each region of a disk is
> independent from the others and so mode changes only impact new
> regions until you do a full rebalance to rewrite everything.  Of
> course, zfs has its own advantages.
>

This drive is formatted with ext4.  It doesn't have LVM or anything just
straight ext4.  Given it is external, I didn't see the point of having
LVM on it and adding another layer to deal with when there is no
benefits to it.  While it does delete some files and overwrite others,
it mostly adds new files.  I'd guess it just throws them on the end but
who knows what it is really doing under the hood that neither of us
knows about.  ;-)

I did notice that is has reached 80% full.  I'm going to start figuring
out what not to backup and such pretty soon.  If I have my videos, I'm
good.  I could backup Documents and some other OS related stuff on
another drive.  I usually backup /etc and the world file.  Hmmm, may
want to grab my local overlay too.  I hadn't thought about that.  Funny
how typing in a email makes me think of things like that. 

Maybe when I do a large backup next time, I'll think to save the output
so I can share it.  That might shed some light on the situation.  I just
thought it was interesting that when it hit about 50 or 60GBs of data it
got a good deal slower and then did the bumpy thing a lot longer too. 
To me, I figured it ran out of PMR space and was slinging stuff good
trying to catch up.  There was several files where it just sat there,
for many seconds with nothing moving.  Usually, 10 seconds is about the
longest wait but it is a large file, movie or something that is GBs or
so.  I'd guess close to a minute for some this last time.  Big
difference.  Most files were around 300MBs too.  Found a source for good
HD stuff that isn't to large.  ;-)

Anyway, thought it worth mentioning.  The drive serves its purpose well
enough for what it does.  Still wouldn't want it somewhere more critical
tho.  Sticking with PMR/CMR until they get things sorted out better.  If
I bought another drive just for external backup use tho, I might get a
SMR if the price was right.  It wouldn't be my first choice tho.

Dale

:-)  :-) 

Re: [gentoo-user] External hard drive and idle activity

[Rich Freeman]

On Mon, Aug 3, 2020 at 8:24 AM Dale  wrote:
>
> In the past, I've never seen the drive on the larger files be that slow even 
> toward the end.  Generally, it stays pretty close to 180MBs/sec or so which 
> is what I usually get with PMR drives.

Yeah, just hard to be certain without ditching the filesystem layer or
doing some kind of comparison.  The difference in write speed across
the drive on recent drives I've gotten is more pronounced than I've
seen on other drives, but these drives tend to be around 12TB.

Definitely the thing to watch out for is a big drop in transfer rate
once a large number of blocks have been transferred continuously, and
then performance returns after you let the drive thrash for a while.
I've seen complaints of zfs rebuilds going from hours/days to
weeks/months in length, so it isn't just a 50% drop when you're doing
worst-case access patterns.  On the other hand I hear that mdadm isn't
so bad, so if the writes are sequential the drive might be better at
skipping the cache, and maybe zfs just does its rebuild
non-sequentially (which isn't really ideal anyway).

I haven't really dug into the guts of how zfs metadata works, but with
btrfs I believe the chunks are basically their own layer, and the
filesystem can scrub them without really any care about what files are
stored in them.  That allows them to be easily scrubbed sequentially.
When I did rebuilds on btrfs they tended to run at about the max
throughput of the drives as long as there wasn't any other access
going on.  It can also do read-only scrubs to check data integrity
sequentially across the disk, which suggests the checksums are stored
at a lower layer and so the data can be verified without worrying
about file fragmentation and so on.  This layering also lets btrfs
switch "RAID modes" on the fly with half of the disk being RAID1 and
half the disk being RAID5 and so on - each region of a disk is
independent from the others and so mode changes only impact new
regions until you do a full rebalance to rewrite everything.  Of
course, zfs has its own advantages.

-- 
Rich

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Rich Freeman wrote:
> On Mon, Aug 3, 2020 at 1:30 AM Dale  wrote:
>> Little update here.  Rich, I think you mentioned it would slow down when it 
>> ran out of PMR space while trying to redo the shingled part.  Up until now, 
>> I hadn't ran into that issue.  It seems the PMR section for this drive is 
>> somewhere around 40 or 50GBs, maybe 60GBs.  I hadn't had time for backups in 
>> over a week so it was a good bit larger than usual.  It was around 70GBs, 
>> maybe 75.  When it got close to the end of the rsync process, I noticed it 
>> slowed down quite a bit.  I'd guess about half or so.  Usually it runs at 
>> around 180 to 190MBs/sec for larger files.  Pretty close to the end, rsync 
>> was showing around 100MBs/sec at best.  It was a little over on some but 
>> mostly a little below that.  Earlier in the process, it was the normal speed.
> I doubt this particular drop is the result of SMR, assuming 100MB/s is
> the instantaneous speed.  100MB/s is still reasonable for a hard drive
> - on newer CMR drives I've seen the speed of dd drop from 200MB/s to
> 100MB/s for sequential writes as the heads move from one end of the
> drive to the other, and then it goes back up to 200MB/s if you start
> over at the beginning (badblocks testing and so on).
>
> That level of drop is probably more likely to be due to filesystem
> overhead and so on - fragmentation/etc.  When SMR buffer overrun
> occurs you REALLY hit a wall and the rates drop quite a bit more than
> that.  If it were a difference of only 50% most would probably
> tolerate it.
>
> Now, if 100MB/s were an updating average across the entire run then
> that would be a different matter, because that would mean that it was
> running at 200MB/s for most of the run, and then probably dropping
> much closer to 0 for a while so as to drive the overall average down
> to 100MB/s.  I'm not sure where you're getting those numbers from so I
> don't know what period that 100MB/s reflects.  For an instantaneous
> speed I'd consider it a completely reasonable performance for a
> typical hard drive when you're writing to a filesystem.  If you were
> using dd or maybe copies of very large files on an efficient
> filesystem you would get better results.
>


I use the --progress option with rsync.  It shows the copy speed,
elapsed time etc for each file.  Some files are small and the speed it
shows isn't accurate at all because the files are just to tiny to get a
accurate measure.  I ignore that info on the smaller files.  Videos
however are larger files and sometimes can take a lot longer to copy
over.  Those tend to be more accurate.  Anyway, that is where the
numbers came from.  I wish I had saved the output but sadly I had
finished my OS updates and needed to logout and back in again.

In the past, I've never seen the drive on the larger files be that slow
even toward the end.  Generally, it stays pretty close to 180MBs/sec or
so which is what I usually get with PMR drives.  Of course, the PMR
drives don't keep doing the bumpy thing for a while when it is done
either.  Maybe it is something else but it sure did the bumpy thing a
lot longer this time. 

Either way, just wanted to update that a large copy made it slow down. 
Other than the initial copy, this is the largest rsync I've ever done to
that drive.  Most are around 20, maybe 25GBs. 

Dale

:-)  :-) 

Re: [gentoo-user] External hard drive and idle activity

[Rich Freeman]

On Mon, Aug 3, 2020 at 1:30 AM Dale  wrote:
>
> Little update here.  Rich, I think you mentioned it would slow down when it 
> ran out of PMR space while trying to redo the shingled part.  Up until now, I 
> hadn't ran into that issue.  It seems the PMR section for this drive is 
> somewhere around 40 or 50GBs, maybe 60GBs.  I hadn't had time for backups in 
> over a week so it was a good bit larger than usual.  It was around 70GBs, 
> maybe 75.  When it got close to the end of the rsync process, I noticed it 
> slowed down quite a bit.  I'd guess about half or so.  Usually it runs at 
> around 180 to 190MBs/sec for larger files.  Pretty close to the end, rsync 
> was showing around 100MBs/sec at best.  It was a little over on some but 
> mostly a little below that.  Earlier in the process, it was the normal speed.

I doubt this particular drop is the result of SMR, assuming 100MB/s is
the instantaneous speed.  100MB/s is still reasonable for a hard drive
- on newer CMR drives I've seen the speed of dd drop from 200MB/s to
100MB/s for sequential writes as the heads move from one end of the
drive to the other, and then it goes back up to 200MB/s if you start
over at the beginning (badblocks testing and so on).

That level of drop is probably more likely to be due to filesystem
overhead and so on - fragmentation/etc.  When SMR buffer overrun
occurs you REALLY hit a wall and the rates drop quite a bit more than
that.  If it were a difference of only 50% most would probably
tolerate it.

Now, if 100MB/s were an updating average across the entire run then
that would be a different matter, because that would mean that it was
running at 200MB/s for most of the run, and then probably dropping
much closer to 0 for a while so as to drive the overall average down
to 100MB/s.  I'm not sure where you're getting those numbers from so I
don't know what period that 100MB/s reflects.  For an instantaneous
speed I'd consider it a completely reasonable performance for a
typical hard drive when you're writing to a filesystem.  If you were
using dd or maybe copies of very large files on an efficient
filesystem you would get better results.

-- 
Rich

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Howdy all,

Little update here.  Rich, I think you mentioned it would slow down when
it ran out of PMR space while trying to redo the shingled part.  Up
until now, I hadn't ran into that issue.  It seems the PMR section for
this drive is somewhere around 40 or 50GBs, maybe 60GBs.  I hadn't had
time for backups in over a week so it was a good bit larger than usual. 
It was around 70GBs, maybe 75.  When it got close to the end of the
rsync process, I noticed it slowed down quite a bit.  I'd guess about
half or so.  Usually it runs at around 180 to 190MBs/sec for larger
files.  Pretty close to the end, rsync was showing around 100MBs/sec at
best.  It was a little over on some but mostly a little below that. 
Earlier in the process, it was the normal speed.

I might add, even tho the copy process has been done for a while now, 20
minutes or more, I can still feel that little bumpy thing it does.  It
seems to have stopped while typing that in.  Unless it is taking a break
and starts up again.  ;-) 

Still, for this use case, it works OK.  I wouldn't want SMR on my /home
or some other partition that needs to be fairly fast at all times.  For
windoze users, well, they used to that slow down and freezes so they
wouldn't notice the difference.  ROFL 

Dale

:-)  :-) 

Re: [gentoo-user] External hard drive and idle activity

[Mick]

On Monday, 6 January 2020 13:53:41 GMT Rich Freeman wrote:
> On Mon, Jan 6, 2020 at 8:25 AM Mick  wrote:
> > If they are used as normal PC drives for regular writing
> > of data, or with back up commands which use rsync, cp, etc. then the disk
> > will fail much sooner than expected because of repeated multiple areas
> > being deleted, before each smaller write.  I recall reading about how
> > short the life of SMR drives was shown to be when used in NAS devices -
> > check google or youtube if you're interested in the specifics.
> 
> Can you give a link - I'm not finding anything, and I'm a bit dubious
> of this claim, because they still are just hard drives.  These aren't
> SSDs and hard drives should not have any kind of erasure limit.

This (random) link strongly recommends against usage in NAS, but gives no 
reliability data:

https://www.storagereview.com/seagate_archive_hdd_review_8tb

This is a youtube video where someone was comparing SMR failures on a NAS: 

https://www.youtube.com/watch?v=CR_bfbOTY1o


> Now, an SMR used for random writes is going to be a REALLY busy drive,
> so I could see the drive being subject to a lot more wear and tear.
> I'm just not aware of any kind of serious study.  And of course any
> particular model of hard drive can have reliability issues (just look
> up the various reliability studies).

Right, I haven't seen any lab reliability studies published.  I would think 
more information could be sourced in IRC/ML where datacenter sysadmins hide to 
compare their ... hardware.  :-)

Reading another random link it seems Dale's 8TB SMR drive has a 20GB 
conventional PMR platter/area in it to catch and cache any small writes.  The 
firmware will subsequently transfer the cached data on the SMR area of the 
drive in due course, after it deletes the requisite adjacent overlapping 
tracks.  This means up to 20GB of initial writes will be normal, dropping to 
lower speeds thereafter as the PMR cache needs to be flushed:

https://www.ixsystems.com/community/threads/smr-hard-drives-do-you-think-they-are-proper-nas-drives.35805/

If this is so, it explains Dale's observation of a hyperactive disk, well 
after it was dismounted.  Its firmware's been busy!

[snip ...]

> Granted, I don't rewrite it often but unless zfs is
> SMR-aware it is still going to be writing lots of modest-sized files
> as the original files get chunked up and distributed across the nodes.
> On the disk lizardfs data just looks like a browser cache, with
> everything in numbered files about 60MB in size in my case.  The files
> also appear to turn over a bit during rebalancing.

I would think bit flipping between the 20GB PMR cache and the 8TB SMR tracks 
represents an increased risk, vis A vis a single-step data transfer.  Data 
scrubbing well after the write has completed and committed to the SMR tracks 
would reveal any anomalies.

What would seriously mess things up is creating a raid with mixed PMR and SMR 
disks and running big (bigger than the internal cache) data writes.  Some PMR 
disks will complete well before the SMR.  I/O blocking and timeouts could 
ensue and the applications performing the writing could hang/fail.

Anyway, write once - read often, fits well the use case for these disks.  They 
should be right at home for long term video and media storage.
-- 
Regards,
Mick

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] External hard drive and idle activity

[Rich Freeman]

On Mon, Jan 6, 2020 at 10:05 AM Dale  wrote:
>
> The drive I have is likely
> done on the drive itself, device managed, which is good for me.

Really the ideal situation are the Host Aware drives.  I have no idea
what percentage of the markets they make.  They fall back to being
device managed if the host doesn't do anything to manage them.

Your drive is device managed.  See the link I posted earlier to the mfr info.

> I've been noticing that too.  Only bad thing is, I can't always tell
> what is in the enclosure.  Sometimes the info is given but sometimes
> not.  I've also seen a few people complain that what they got was not
> the model of drive they thought.

Yeah, there are no guarantees as to what you'll get if you go the
shucking route.  If you absolutely need a certain amount of cache or a
red firmware then you're just going to have to pay double to get that
guarantee.

For my application I'd definitely prefer the red firmware, but it
isn't really the end of the world if the drive takes a few seconds to
timeout on that one failure every 5 years.  I'm not going to pay
double just to guarantee a particular model.  If it were $20 more that
would be another matter, but we're talking $180 vs $350 here.

The other gotcha is that if you want to do a warranty replacement at
some point you're going to have to put it back in the enclosure to do
so.  That means hanging onto enclosures, and is of course a bit of a
pain besides.  Again, with a 50% reduction in cost you'd need to have
a lot of drive failures to be worth worrying about, especially since I
believe drive warranties are getting shorter anyway.

If you're in a typical enterprise situation then you're going to want
to just buy the bare drives with the standard warranty/etc.  If you
buy in bulk chances are you're getting a discount anyway (and then
maybe you can get them in caddies or whatever).  The Best Buy deals
are sporadic anyway and limited in quantity - you could never run a
data center that way.  That's why they're priced the way they are...

-- 
Rich

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Rich Freeman wrote:
> On Mon, Jan 6, 2020 at 9:18 AM Dale  wrote:
>> Rich Freeman wrote:
>>> On Mon, Jan 6, 2020 at 8:25 AM Mick  wrote:
 If they are used as normal PC drives for regular writing
 of data, or with back up commands which use rsync, cp, etc. then the disk 
 will
 fail much sooner than expected because of repeated multiple areas being
 deleted, before each smaller write.  I recall reading about how short the 
 life
 of SMR drives was shown to be when used in NAS devices - check google or
 youtube if you're interested in the specifics.
>>> Can you give a link - I'm not finding anything, and I'm a bit dubious
>>> of this claim, because they still are just hard drives.  These aren't
>>> SSDs and hard drives should not have any kind of erasure limit.
>>>
>>> Now, an SMR used for random writes is going to be a REALLY busy drive,
>>> so I could see the drive being subject to a lot more wear and tear.
>>> I'm just not aware of any kind of serious study.  And of course any
>>> particular model of hard drive can have reliability issues (just look
>>> up the various reliability studies).
>>>
>> I ran up on this article however, it is a short time frame.  Still might
>> be a interesting read tho.
>>
>> https://blogs.dropbox.com/tech/2019/07/smr-what-we-learned-in-our-first-year/
> That article makes no mention of reliability issues with SMR.  In
> fact, they mention that they want 40% of their storage to be on SMR by
> now.  Clearly they wouldn't be doing that if the drives failed
> frequently.
>
> Note that they did modify their software to have write patterns
> suitable for SMR.  That is the key here.  You absolutely have to
> engineer your application to be suitable for SMR, or only choose SMR
> if your application is already suitable.  You can't just expect these
> drives to perform remotely acceptably if you just throw random writes
> at them.

True but they likely have the drives that have it handled in software,
host managed I think they call it.  Another article I read was talking
about three different approaches to SMR.  The drive I have is likely
done on the drive itself, device managed, which is good for me.  The
ones in the article appear to manage the data transfer in software.  I
noticed they also use SSDs as sort of a temporary storage, if I
understood that correctly.  I think they did that to speed things up a bit.


>> I'm still a bit curious and somewhat untrusting of those things tho.
>> Regular hard drives go bad often enough as it is.  We don't need some
>> fancy unknown thing inserted just to add more issues.  Sort of reminds
>> me of the init thingy.  Each thing added is another failure point.
> Obviously they're relatively new, but they seem reliable enough.
> They're just not suitable for general purpose use.
>
>> I'm going to test my ebay skills and see if I can find some non-SMR
>> drives.  It sounds like some require some research to know if they are
>> or not.  :/
> That's pretty simple.  Find a drive that looks reasonable
> price/capacity/etc-wise.  Then just google the model number to confirm
> it isn't SMR.
>
> If you're in the US though you're probably best off shucking drives
> from Best Buy these days.  A drive that costs $350 as a bare drive
> will get sold for $180 in a USB enclosure.  I think it is just market
> segmentation.  They want to get top dollar from enterprise users, and
> they aren't going to be shucking drives from Best Buy bought on "limit
> 1 item per customer" sales.  By shucking I'm getting 12TB red drives
> for less than the cost of a 6TB green drive.  Just be aware that if
> your PSU is old you'll need to tape over some of the SATA power pins.
> New PSUs - even cheap ones - haven't given me any trouble.
>
> I'm sure there are more up-to-date guides as these days the drives are
> 12TB, but here is the gist of it:
> https://www.reddit.com/r/DataHoarder/comments/7fx0i0/wd_easystore_8tb_compendium/
>
> If you aren't in the US I have no idea whether equivalent deals are
> available.  That subreddit is a good place to go for info on cheap
> hard drives though.
>


I've been noticing that too.  Only bad thing is, I can't always tell
what is in the enclosure.  Sometimes the info is given but sometimes
not.  I've also seen a few people complain that what they got was not
the model of drive they thought. 

I suspect one can get a adapter for that P/S connector.  I can't recall
when I got the P/S I currently have but it is a few years old.  I think
it's a ThermalTake or something like that.  I got to overclockers forum
where they list good ones.  I'm almost certain it has standard
connectors which may be a problem.  I've read about having to cover up a
pin or something but never seen one in person. 

This is a educational thread. I didn't even know SMR was a thing until
this thread came along. 

Dale

:-)  :-) 

Re: [gentoo-user] External hard drive and idle activity

[Rich Freeman]

On Mon, Jan 6, 2020 at 9:18 AM Dale  wrote:
>
> Rich Freeman wrote:
> > On Mon, Jan 6, 2020 at 8:25 AM Mick  wrote:
> >> If they are used as normal PC drives for regular writing
> >> of data, or with back up commands which use rsync, cp, etc. then the disk 
> >> will
> >> fail much sooner than expected because of repeated multiple areas being
> >> deleted, before each smaller write.  I recall reading about how short the 
> >> life
> >> of SMR drives was shown to be when used in NAS devices - check google or
> >> youtube if you're interested in the specifics.
> > Can you give a link - I'm not finding anything, and I'm a bit dubious
> > of this claim, because they still are just hard drives.  These aren't
> > SSDs and hard drives should not have any kind of erasure limit.
> >
> > Now, an SMR used for random writes is going to be a REALLY busy drive,
> > so I could see the drive being subject to a lot more wear and tear.
> > I'm just not aware of any kind of serious study.  And of course any
> > particular model of hard drive can have reliability issues (just look
> > up the various reliability studies).
> >
>
> I ran up on this article however, it is a short time frame.  Still might
> be a interesting read tho.
>
> https://blogs.dropbox.com/tech/2019/07/smr-what-we-learned-in-our-first-year/

That article makes no mention of reliability issues with SMR.  In
fact, they mention that they want 40% of their storage to be on SMR by
now.  Clearly they wouldn't be doing that if the drives failed
frequently.

Note that they did modify their software to have write patterns
suitable for SMR.  That is the key here.  You absolutely have to
engineer your application to be suitable for SMR, or only choose SMR
if your application is already suitable.  You can't just expect these
drives to perform remotely acceptably if you just throw random writes
at them.

> I'm still a bit curious and somewhat untrusting of those things tho.
> Regular hard drives go bad often enough as it is.  We don't need some
> fancy unknown thing inserted just to add more issues.  Sort of reminds
> me of the init thingy.  Each thing added is another failure point.

Obviously they're relatively new, but they seem reliable enough.
They're just not suitable for general purpose use.

> I'm going to test my ebay skills and see if I can find some non-SMR
> drives.  It sounds like some require some research to know if they are
> or not.  :/

That's pretty simple.  Find a drive that looks reasonable
price/capacity/etc-wise.  Then just google the model number to confirm
it isn't SMR.

If you're in the US though you're probably best off shucking drives
from Best Buy these days.  A drive that costs $350 as a bare drive
will get sold for $180 in a USB enclosure.  I think it is just market
segmentation.  They want to get top dollar from enterprise users, and
they aren't going to be shucking drives from Best Buy bought on "limit
1 item per customer" sales.  By shucking I'm getting 12TB red drives
for less than the cost of a 6TB green drive.  Just be aware that if
your PSU is old you'll need to tape over some of the SATA power pins.
New PSUs - even cheap ones - haven't given me any trouble.

I'm sure there are more up-to-date guides as these days the drives are
12TB, but here is the gist of it:
https://www.reddit.com/r/DataHoarder/comments/7fx0i0/wd_easystore_8tb_compendium/

If you aren't in the US I have no idea whether equivalent deals are
available.  That subreddit is a good place to go for info on cheap
hard drives though.

-- 
Rich

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Rich Freeman wrote:
> On Mon, Jan 6, 2020 at 8:25 AM Mick  wrote:
>> If they are used as normal PC drives for regular writing
>> of data, or with back up commands which use rsync, cp, etc. then the disk 
>> will
>> fail much sooner than expected because of repeated multiple areas being
>> deleted, before each smaller write.  I recall reading about how short the 
>> life
>> of SMR drives was shown to be when used in NAS devices - check google or
>> youtube if you're interested in the specifics.
> Can you give a link - I'm not finding anything, and I'm a bit dubious
> of this claim, because they still are just hard drives.  These aren't
> SSDs and hard drives should not have any kind of erasure limit.
>
> Now, an SMR used for random writes is going to be a REALLY busy drive,
> so I could see the drive being subject to a lot more wear and tear.
> I'm just not aware of any kind of serious study.  And of course any
> particular model of hard drive can have reliability issues (just look
> up the various reliability studies).
>

I ran up on this article however, it is a short time frame.  Still might
be a interesting read tho.

https://blogs.dropbox.com/tech/2019/07/smr-what-we-learned-in-our-first-year/

I'm still a bit curious and somewhat untrusting of those things tho. 
Regular hard drives go bad often enough as it is.  We don't need some
fancy unknown thing inserted just to add more issues.  Sort of reminds
me of the init thingy.  Each thing added is another failure point. 

I'm going to test my ebay skills and see if I can find some non-SMR
drives.  It sounds like some require some research to know if they are
or not.  :/

Dale

:-)  :-) 

Re: [gentoo-user] External hard drive and idle activity

[Rich Freeman]

On Mon, Jan 6, 2020 at 8:25 AM Mick  wrote:
>
> If they are used as normal PC drives for regular writing
> of data, or with back up commands which use rsync, cp, etc. then the disk will
> fail much sooner than expected because of repeated multiple areas being
> deleted, before each smaller write.  I recall reading about how short the life
> of SMR drives was shown to be when used in NAS devices - check google or
> youtube if you're interested in the specifics.

Can you give a link - I'm not finding anything, and I'm a bit dubious
of this claim, because they still are just hard drives.  These aren't
SSDs and hard drives should not have any kind of erasure limit.

Now, an SMR used for random writes is going to be a REALLY busy drive,
so I could see the drive being subject to a lot more wear and tear.
I'm just not aware of any kind of serious study.  And of course any
particular model of hard drive can have reliability issues (just look
up the various reliability studies).

> Personally, I would only use such a drive for 'keepers'.  Say, films I intend
> to write once and watch many times, ripped music albums, family photos, etc.
> For OS files and other temporary backups I would use a normal PC drive.

Certainly I would never use an SMR for an OS or /home.  Backups should
be fine, as long as you're using a sequential backup file format.
tar/duplicity should be fine.  Dar is probably fine but I'd need to
check (I think it just writes the index to the end, so the seeking
issues are on reading and not writing).  Even zip/etc is going to be
fine.  What is going to be a problem is anything that just replicates
the original data as all the separate files/directories that exist on
the original drive, like rsync/rsnapshot/etc.  Those formats are of
course attractive because the backup is just a replica of the
original, but they involve random writes.  Most formats that just
create a bunch of files named archive-001.foo that need a special
command to restore are going to be fine.

I personally haven't encountered a need to consider an SMR drive as
you can shuck those 12TB Easystore drives for something like $180 on
sale, at least in the US.  Those are just standard drives (often with
red firmware).  I couldn't even use them for my multimedia as I'm
storing that stuff on lizardfs right now and that breaks everything
into chunks.  Granted, I don't rewrite it often but unless zfs is
SMR-aware it is still going to be writing lots of modest-sized files
as the original files get chunked up and distributed across the nodes.
On the disk lizardfs data just looks like a browser cache, with
everything in numbered files about 60MB in size in my case.  The files
also appear to turn over a bit during rebalancing.

-- 
Rich

Re: [gentoo-user] External hard drive and idle activity

[Mick]

On Monday, 6 January 2020 08:48:13 GMT Stefan Schmiedl wrote:
> Dale,
> 
> "Dale" , 06.01.2020, 09:29:
> > Also, when looking for a drive to buy, what should one look at to see if
> > it is a SMR drive?

You will need to visit the OEMs website and dig into the documentation they 
provide.  Keywords like "archive drive/disk/format", "shingled magnetic 
recording" and "SMR", would be a giveaway this is not a normal PC drive.


> > While it may be OK for my backups, I'd like to avoid
> > them on the drives inside my rig that are used for the OS or /home.  I
> > dunno, just a gut thing.
> 
> it's not "just a gut thing". SMR drives are not meant for random
> access writing; they write like a tape and read like a disk.
> 
> A while ago, one of my clients bought one of those things
> to replace an older failing backup drive. The next night, the
> backup took hours instead of minutes. No knowing what was inside
> the box, I did some measurements and discovered that the first
> few files were written quickly, then things got really slow,
> with the rsync process waiting (state "D") for the drive to
> finish.
> 
> tar-based backups went much quicker, though, which matches the
> expected behaviour of SMR drives; the drive did not need to rewrite
> many large areas due to many small changes, instead it only had to
> write one large area due to one large change.
> 
> s.

Stefan reinforced a point made earlier by Richard (I think).  These drives are 
only good for linear backups, like tar performs when it appends newer files to 
an existing tarball.  If they are used as normal PC drives for regular writing 
of data, or with back up commands which use rsync, cp, etc. then the disk will 
fail much sooner than expected because of repeated multiple areas being 
deleted, before each smaller write.  I recall reading about how short the life 
of SMR drives was shown to be when used in NAS devices - check google or 
youtube if you're interested in the specifics.

Personally, I would only use such a drive for 'keepers'.  Say, films I intend 
to write once and watch many times, ripped music albums, family photos, etc.  
For OS files and other temporary backups I would use a normal PC drive.

PS. When you put together a tar script do not forget to add --xattrs.  If not, 
you'll find some commands break when you run them from a restored fs.
-- 
Regards,
Mick

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] External hard drive and idle activity

[Stefan Schmiedl]

Dale,

"Dale" , 06.01.2020, 09:29:

> Also, when looking for a drive to buy, what should one look at to see if
> it is a SMR drive?  While it may be OK for my backups, I'd like to avoid
> them on the drives inside my rig that are used for the OS or /home.  I
> dunno, just a gut thing. 

it's not "just a gut thing". SMR drives are not meant for random 
access writing; they write like a tape and read like a disk.

A while ago, one of my clients bought one of those things 
to replace an older failing backup drive. The next night, the
backup took hours instead of minutes. No knowing what was inside
the box, I did some measurements and discovered that the first
few files were written quickly, then things got really slow,
with the rsync process waiting (state "D") for the drive to
finish.

tar-based backups went much quicker, though, which matches the
expected behaviour of SMR drives; the drive did not need to rewrite
many large areas due to many small changes, instead it only had to
write one large area due to one large change.

s.

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Dale wrote:
> <<< SNIP >>>
> Should I change the mounting options for this drive?  I've read some
> people use certain options for SSD and they are needed.  This is the
> options being used according to mount:
>
> /dev/sdj1 on /run/media/dale/8tb-backup type ext4
> (rw,nosuid,nodev,relatime,uhelper=udisks2)
>
> That I guess is the default way KDE/udisks/etc does it. 
>
> One other question, what should I look for to avoid these types of
> drives?  I'll go back and look but I don't recall it saying anything
> about this.  I'm also trying to figure out if this is a good thing or not. 
>
> Thanks much for the info.  I'll read this one again shortly. Lot of info
> to absorb. 
>
> Dale
>
> :-)  :-) 
>


Finally got around to rebooting.  It was a experience for sure.  X
wouldn't come up, the plasma thingy doing its thing.  Jeepers.  No
wonder I hate rebooting.  ROFL  I enabled the options mentioned in the
kernel and finally rebooted using it. This is the option and the current
mount info:


root@fireball / # zcat /proc/config.gz | grep DM_ZONED
CONFIG_DM_ZONED=y
root@fireball / #

/dev/sdj1 on /run/media/dale/8tb-backup type ext4
(rw,nosuid,nodev,relatime,uhelper=udisks2)


It looks the same to me.  While it seems safe as is, should I be
changing something to make it use either additional or other options? 

While it is a backup, I'd like to make sure they are worth something if
the need should arise. 

Also, when looking for a drive to buy, what should one look at to see if
it is a SMR drive?  While it may be OK for my backups, I'd like to avoid
them on the drives inside my rig that are used for the OS or /home.  I
dunno, just a gut thing. 

Thanks.

Dale

:-)  :-) 

Re: [gentoo-user] External hard drive and idle activity

[Rich Freeman]

On Fri, Jan 3, 2020 at 8:51 AM Spackman, Chris  wrote:
>
> udisksctl power-off --block-device /dev/sdx
>
> I didn't see that command mentioned in the thread yet. I've been using
> it, after umount, for about 8 months for roughly weekly backups and
> some misc storage. So far, I've not seen any problems with it. The drive
> immediately shuts down, and there haven't been any data or performance
> issues.
>
> But because no one else has mentioned it, I wonder udisksctl is not the
> best tool in this case?
>

I suspect that running this command will either not power off the
drive in this case, or if it does it will have the exact same impact
as pulling the plug.  If the drive is unmounted then the kernel has
received guarantees from the drive that all data is written
persistently.  It is unlikely that a drive will lose the data after
this point, but if has some bug then short of a firmware update I
doubt there is any reliable workaround short of leaving it on forever.

-- 
Rich

Re: [gentoo-user] External hard drive and idle activity

[Spackman, Chris]

On 2020/01/01 at 08:00pm, Dale wrote:
> Grant Taylor wrote:
> > On 1/1/20 5:09 PM, Dale wrote:

> > Note:  umount will normally block until buffers are flushed to disk.
> >
> >> Is it safe to turn it off even tho it is doing whatever it is doing?
> >
> > I wouldn't.
> >
> >> Should I wait?
> >
> > I would.
> >
> >> Does it matter?
> >
> > Maybe.

[snip]

> If I touch the enclosure and feel it doing something, I leave it on,
> just in case.  I actually been wondering about this for a while. 
> Sometimes it will stop after a couple minutes, sometimes it is still
> doing its thing 30 minutes later.  In the case of the first, I was
> concerned about files being cached etc.

I have a similar drive (8TB external. WD, iirc). It did something
similar - it would seem to still be in use even after I umounted it -
and I wasn't sure if it was okay to unplug (no physical off
switch). Somewhere I found this command to shut down the drive:

udisksctl power-off --block-device /dev/sdx

I didn't see that command mentioned in the thread yet. I've been using
it, after umount, for about 8 months for roughly weekly backups and
some misc storage. So far, I've not seen any problems with it. The drive
immediately shuts down, and there haven't been any data or performance
issues.

But because no one else has mentioned it, I wonder udisksctl is not the
best tool in this case?


-- 
Chris Spackman  ch...@osugisakae.com

ESL Coordinator The Graham Family of Schools
ESL Instructor  Columbus State Community College
Japan Exchange and Teaching Program   Wajima, Ishikawa 1995-1998
Linux user since 1998 Linux User #137532

Re: [gentoo-user] External hard drive and idle activity

[Rich Freeman]

On Thu, Jan 2, 2020 at 7:43 PM Wol  wrote:
>
> And yes - the 8TB capacity gave it away - I think the largest "normal"
> drives available are 4TB at present ... anything bigger must be shingled.
>

I have several non-SMR drives in the 8-12TB range, shucked from those
WD external drives that occasionally are on sale at Best Buy.  While
the drives inside are white-label people have managed to socially
engineer the support team at WD to reveal that they aren't SMR.

But, you definitely need to check.

-- 
Rich

Re: [gentoo-user] External hard drive and idle activity

[Wol]

On 02/01/2020 19:12, Rich Freeman wrote:

For reads they're completely normal.  For sequential writes to unused
space they're completely normal.  For random writes or overwrites they
are significantly different from traditional hard drives.


That's why the "S" stands for "shingled" (which means "overlapped", 
think of a shingled roof - which is your normal overlapping tile jobbie).


The point of a shingled drive is that it has a narrow read head - let's 
call it one unit wide. But a write head is larger, lets call it three 
units wide. If each track is one unit wide - the correct size to be read 
- then writing a track will destroy two tracks.


So the write head lays down data in concentric rings, and CAN'T 
overwrite data - it has to wipe everything and rewrite it. So as 
mentioned before you really do not want these drives for random writes - 
think of them as a bit like a random-write tape drive ie you can get 
away with it but you're better off not trying.


And yes - the 8TB capacity gave it away - I think the largest "normal" 
drives available are 4TB at present ... anything bigger must be shingled.


Cheers,
Wol

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Rich Freeman wrote:
> On Thu, Jan 2, 2020 at 1:41 PM Dale  wrote:
>> Rich Freeman wrote:
>>> Out of curiosity, what model drive is it?  Is it by chance an SMR /
>>> archive drive?
>> Device Model: ST8000AS0003-2HH188
>>
>> I recall reading about SMR but can't recall the details of what it is.
>> As far as I know, this is just a basic 8TB drive.
> This is an SMR drive.  You should DEFINITELY read up on what they are.
>
> For reads they're completely normal.  For sequential writes to unused
> space they're completely normal.  For random writes or overwrites they
> are significantly different from traditional hard drives.
>
> They work a bit like an SSD in the sense that blocks are arranged into
> larger erase regions.  Within a region blocks can only be written
> sequentially.  If you want to overwrite one block in the middle of a
> region, the drive will read the entire region into RAM, then write the
> entire region sequentially with the overwritten block to a new spot on
> the disk.  This is just like in an SSD where if try to overwrite a
> block in a region with any unTRIMmed blocks the drive must read the
> entire region, erase the region, and write the modified region.
>
> Except that in an SSD those extra reads/writes operate with SSD access
> times.  With an SMR drive those extra reads/writes operate with hard
> drive latencies, so they're MUCH more costly.
>
> For backup use they're usually fine, IF you're writing in a sequential
> file format that is appended to.  If you're using rsync to do your
> backups then that isn't what you're doing and you're probably paying a
> heavy penalty.  If you were doing incremental backups using
> tar/duplicity/whatever then you'd probably be fine.
>
> Some filesystems might be optimized for these drives to reduce the
> amount of overwriting in place.  I haven't looked into it.  I'd expect
> a log-based filesystem to work fairly well, though those can have high
> levels of fragmentation which is better suited for SSD than SMR.
>
> These drives all have fairly active firmware that manages this special
> overwrite process so that they can be used with operating systems that
> are naive to how they work.  I wouldn't be surprised if this is what
> is causing the drive to be active after you unmount it.  In theory it
> should be harmless to power it off.  However, leaving it powered on
> probably will improve its performance as it can take care of any
> garbage collection before the next time you use it.  If whatever
> journal it is using to speed things up gets full then you'll feel the
> full brunt of any write penalties until it is flushed.
>
> You might want to seriously consider changing to a backup format that
> just creates big tail-appended files containing incremental changes.
> Something like rsync that just outputs bazillions of little files is
> going to create lots of random writes when things change, vs
> consolidating all those changes into one file that just grows at the
> end.  Treat them the way you would a tape (which is what tar was
> designed for).
>
> Nothing wrong with SMR drives per se - they can potentially be cheaper
> especially for backup (using an appropriate file format), and are just
> as fast for reading so they're also great for infrequently changing
> bulky data.  However, random writes are very costly and you should be
> aware of that going in...
>


I had no idea this was that type of drive.  I wonder if any of my other
drives are this type as well.  If it matters, I use ext4 on my drives,
except /boot, but I've thought about using something that is also
compatible with windoze, just in case I need to copy some things to
someone else's system.  Thing is, I'm not real big on those file systems
since I lose some info such as permissions and all.  Does my using ext4
change this any?  Better or worse?

Should I change the mounting options for this drive?  I've read some
people use certain options for SSD and they are needed.  This is the
options being used according to mount:

/dev/sdj1 on /run/media/dale/8tb-backup type ext4
(rw,nosuid,nodev,relatime,uhelper=udisks2)

That I guess is the default way KDE/udisks/etc does it. 

One other question, what should I look for to avoid these types of
drives?  I'll go back and look but I don't recall it saying anything
about this.  I'm also trying to figure out if this is a good thing or not. 

Thanks much for the info.  I'll read this one again shortly. Lot of info
to absorb. 

Dale

:-)  :-) 

Re: [gentoo-user] External hard drive and idle activity

[madscientistatlarge]

‐‐‐ Original Message ‐‐‐
On Thursday, January 2, 2020 12:12 PM, Rich Freeman  wrote:

> On Thu, Jan 2, 2020 at 1:41 PM Dale rdalek1...@gmail.com wrote:
>
> > Rich Freeman wrote:
> >
> > > Out of curiosity, what model drive is it? Is it by chance an SMR /
> > > archive drive?
> >
> > Device Model: ST8000AS0003-2HH188
> > I recall reading about SMR but can't recall the details of what it is.
> > As far as I know, this is just a basic 8TB drive.
>
> This is an SMR drive. You should DEFINITELY read up on what they are.
>
> For reads they're completely normal. For sequential writes to unused
> space they're completely normal. For random writes or overwrites they
> are significantly different from traditional hard drives.
>
> They work a bit like an SSD in the sense that blocks are arranged into
> larger erase regions. Within a region blocks can only be written
> sequentially. If you want to overwrite one block in the middle of a
> region, the drive will read the entire region into RAM, then write the
> entire region sequentially with the overwritten block to a new spot on
> the disk. This is just like in an SSD where if try to overwrite a
> block in a region with any unTRIMmed blocks the drive must read the
> entire region, erase the region, and write the modified region.
>
> Except that in an SSD those extra reads/writes operate with SSD access
> times. With an SMR drive those extra reads/writes operate with hard
> drive latencies, so they're MUCH more costly.
>
> For backup use they're usually fine, IF you're writing in a sequential
> file format that is appended to. If you're using rsync to do your
> backups then that isn't what you're doing and you're probably paying a
> heavy penalty. If you were doing incremental backups using
> tar/duplicity/whatever then you'd probably be fine.
>
> Some filesystems might be optimized for these drives to reduce the
> amount of overwriting in place. I haven't looked into it. I'd expect
> a log-based filesystem to work fairly well, though those can have high
> levels of fragmentation which is better suited for SSD than SMR.
>
> These drives all have fairly active firmware that manages this special
> overwrite process so that they can be used with operating systems that
> are naive to how they work. I wouldn't be surprised if this is what
> is causing the drive to be active after you unmount it. In theory it
> should be harmless to power it off. However, leaving it powered on
> probably will improve its performance as it can take care of any
> garbage collection before the next time you use it. If whatever
> journal it is using to speed things up gets full then you'll feel the
> full brunt of any write penalties until it is flushed.

> Rich

Thank you for the excellent education!  I haven't read the full thread but I'd 
also suggest that running Wireshark on the USB port would likely help diagnose 
any other issues.  I'm having similiar problems with an external drive 
"freeezing" and refusing to unmount normally and this will be my next step to 
diagnose it.

Re: [gentoo-user] External hard drive and idle activity

[Rich Freeman]

On Thu, Jan 2, 2020 at 1:41 PM Dale  wrote:
>
> Rich Freeman wrote:
> >
> > Out of curiosity, what model drive is it?  Is it by chance an SMR /
> > archive drive?
>
> Device Model: ST8000AS0003-2HH188
>
> I recall reading about SMR but can't recall the details of what it is.
> As far as I know, this is just a basic 8TB drive.

This is an SMR drive.  You should DEFINITELY read up on what they are.

For reads they're completely normal.  For sequential writes to unused
space they're completely normal.  For random writes or overwrites they
are significantly different from traditional hard drives.

They work a bit like an SSD in the sense that blocks are arranged into
larger erase regions.  Within a region blocks can only be written
sequentially.  If you want to overwrite one block in the middle of a
region, the drive will read the entire region into RAM, then write the
entire region sequentially with the overwritten block to a new spot on
the disk.  This is just like in an SSD where if try to overwrite a
block in a region with any unTRIMmed blocks the drive must read the
entire region, erase the region, and write the modified region.

Except that in an SSD those extra reads/writes operate with SSD access
times.  With an SMR drive those extra reads/writes operate with hard
drive latencies, so they're MUCH more costly.

For backup use they're usually fine, IF you're writing in a sequential
file format that is appended to.  If you're using rsync to do your
backups then that isn't what you're doing and you're probably paying a
heavy penalty.  If you were doing incremental backups using
tar/duplicity/whatever then you'd probably be fine.

Some filesystems might be optimized for these drives to reduce the
amount of overwriting in place.  I haven't looked into it.  I'd expect
a log-based filesystem to work fairly well, though those can have high
levels of fragmentation which is better suited for SSD than SMR.

These drives all have fairly active firmware that manages this special
overwrite process so that they can be used with operating systems that
are naive to how they work.  I wouldn't be surprised if this is what
is causing the drive to be active after you unmount it.  In theory it
should be harmless to power it off.  However, leaving it powered on
probably will improve its performance as it can take care of any
garbage collection before the next time you use it.  If whatever
journal it is using to speed things up gets full then you'll feel the
full brunt of any write penalties until it is flushed.

You might want to seriously consider changing to a backup format that
just creates big tail-appended files containing incremental changes.
Something like rsync that just outputs bazillions of little files is
going to create lots of random writes when things change, vs
consolidating all those changes into one file that just grows at the
end.  Treat them the way you would a tape (which is what tar was
designed for).

Nothing wrong with SMR drives per se - they can potentially be cheaper
especially for backup (using an appropriate file format), and are just
as fast for reading so they're also great for infrequently changing
bulky data.  However, random writes are very costly and you should be
aware of that going in...

-- 
Rich

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Rich Freeman wrote:
> On Thu, Jan 2, 2020 at 5:49 AM Dale  wrote:
>> lol  I didn't think of that and I don't recall anyone else thinking of
>> it either.
> That is because syncing before unmounting doesn't do anything.  Unless
> you use --lazy umount blocks until all writes are complete to a
> device.  The instant it returns as far as the kernel is concerned the
> device should be safe to power off.
>
> If you do a sync first then of course the umount will complete more
> quickly, since all writes should already be flushed.
>
> I have no idea what your device is doing after it is unmounted, but it
> doesn't have anything to do with the linux kernel unless some process
> is directly accessing the raw device (very unlikely).  Maybe the drive
> firmware is doing some kind of housekeeping, or maybe the drive has
> some kind of vibration in it that just makes it feel like it is doing
> something.  Or maybe the NSA or Red Army has hacked your firmware and
> it is doing who knows what (yes, the NSA bit at least is a thing).  In
> any case, chances are the drive manufacturer has accounted for sudden
> power loss in the design because if they didn't there would be a ton
> of complaints, since there is nothing you can do about this sort of
> thing assuming the firmware is up to something.
>
> Out of curiosity, what model drive is it?  Is it by chance an SMR /
> archive drive?  Due to the limitations on how those write data out I
> could see them implementing an internal filesystem that journals
> incoming data and then writes it back out after the fact.  If so then
> that might happen even after the kernel thinks it is unmounted.
> However, such a drive firmware would probably use a journal that
> ensures data is safe even if power is cut mid-operation.  The drive
> isn't supposed to report that a write is completed until it is
> durable.
>


This is the drive info:


root@fireball / # smartctl -i /dev/sdj
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.40-gentoo] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: ST8000AS0003-2HH188
Serial Number:    WCT0BQ2Y
LU WWN Device Id: 5 000c50 0ac7d172a
Firmware Version: 0003
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate:    5425 rpm
Form Factor:  3.5 inches
Device is:    Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Jan  2 12:27:14 2020 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

root@fireball / #


I recall reading about SMR but can't recall the details of what it is. 
As far as I know, this is just a basic 8TB drive.  I didn't get to fancy
since I knew I wouldn't be running it to much.  Honestly, for this task
most any drive would do. 

I added the sync command to my little script just as a added measure. 
It may not matter but at least I know it should be in sync according to
the kernel.  What the drive does when it gets to it, only the drive knows. 

I might add, it doesn't always have the same feel.  There are times when
I unmount the drive and it just sits there.  I can tell it is spinning
but the heads aren't moving.  Most of the time tho, it has this little
bumpy feel.  It sort of seems random.  It's a lot like it feels when I'm
doing a backup just not nearly as much.  When doing a backup it has that
bumpy feel a lot.  I can feel it on my keyboard even.  Once unmounted,
it still does it but a lot less frequent.  The drive finished a run of
the script while typing the last paragraph.  I've typed this paragraph
and I have not felt a single bump.  It's still mounted even but still no
bumpy feel.  I just unmounted it and I felt a few bumps but then it went
back to idle.  Still no bumpy feel and I'm a bit of a slow typer. 

My biggest confusion, was the files safe?  I just felt a small set of
bumps.  Felt like three or four but back to nothing again.  The lights
on the enclosure didn't change either.  A couple more bumps.  It's weird
because I can never predict when it will do it. 

Things get weird sometimes.  lol  It seems I always run into these weird
things too.  :/ 

Dale

:-)  :-) 

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Francisco Ares wrote:
>
>
> Em qui., 2 de jan. de 2020 às 07:49, Dale  > escreveu:
>
> Neil Bothwick wrote:
> > On Wed, 1 Jan 2020 20:27:34 -0600, Dale wrote:
> >
> >>> If you wait for a few seconds after the backup is completed before
> >>> you unmount the drive, you should be OK.  Although it may slow
> down
> >>> or any LEDs flash less frequently the drive may not stop spinning,
> >>> unless there is some power save process taking control of it.
> >>> 
> >> Given the speed, it is likely done when I tell the KDE thingy to
> >> unmount.  Usually, I start the backup and walk away for a few
> minutes. 
> >> I do it with one of my scripts, if one can call what I do a
> script, and
> >> it does the date command at the end.
> > Add a sync command to the end of the script to make sure all
> filesystem
> > buffers are flushed to disk before it finishes.
> >
> >
>
> Now there's a idea.  Between the sync command and the unmount process,
> it should be safe for sure.  You made good use of those brain cells. 
> lol  I didn't think of that and I don't recall anyone else thinking of
> it either.  Still wonder what the heck it is doing sometimes tho. 
>
> Dale
>
> :-)  :-) 
>
>
> Another thing to be considered is S.M.A.R.T. activity, that is proper
> to the hard drive firmware.  Don't know if it is safe or not to power
> it down during that process though.  I would wait until it really
> finishes all activity.
>
> But do you really need to power it down?  Mechanic (magnetic) hard
> drives are known to have shorter lives if they are turned on and off
> frequently.  My desk computer is rarely turned off, I have 4 hard
> drives, the oldest is close to 10 years old and SMART diagnostics are
> always good.
>
> Best regards,
> Francisco


That's what I meant when talking about selftest or media checks.  As we
know, most all drives have this function nowadays.  As far as I know,
all drives do this now, except SSD anyway.  They may have some other
thing they do tho.

I've thought about the power on/off cycle and it's one reason I only
back up once a day most all the time.  Sometimes, I may not backup for a
couple days or so depending on what's going on.  While my puter runs
24/7, I've had drives go bad in them before too.  Something always
happens whether it is power on/off or media going bad or just plain
mechanical/electrical failure.  It's likely a gamble no matter what one
does.  It's always something to consider tho. 

Dale

:-)  :-) 

Re: [gentoo-user] External hard drive and idle activity

[Rich Freeman]

On Thu, Jan 2, 2020 at 11:23 AM Mick  wrote:
>
> On Thursday, 2 January 2020 14:43:58 GMT Rich Freeman wrote:
>
>
> > Out of curiosity, what model drive is it?  Is it by chance an SMR /
> > archive drive?
>
> Good catch!  I hadn't thought of this - the Linux kernel will need to have
> DM_ZONED enabled I think, for the OS to manager the shingled writes
> sequentially, but I don't have this enabled here because AFAIK I have no such
> drives in my possession.

I haven't looked into the details of how it works.  Certainly the
kernel should be able to optimize writes to such disks and utilize
TRIM if supported by the drive.  But it isn't strictly needed as you
go on to say.

> > Due to the limitations on how those write data out I
> > could see them implementing an internal filesystem that journals
> > incoming data and then writes it back out after the fact.
>
> SMR drives which implement a 'device managed' write mechanism, will use their
> own firmware to control data storage.  The OS would not be aware of anything
> being different to a conventional drive.

Correct.

> > If so then
> > that might happen even after the kernel thinks it is unmounted.
> > However, such a drive firmware would probably use a journal that
> > ensures data is safe even if power is cut mid-operation.  The drive
> > isn't supposed to report that a write is completed until it is
> > durable.
>
> Which I take it to mean the drive would not be unmounted by the OS until it is
> safe to do so and for all intends and purposes it will also be safe to be
> powered down thereafter.

Yes - even if the drive is doing its own data shuffling after being
unmounted, it should still be safe to power off.

In any case, I was just speculating as to why it might be doing writes
when not mounted.  I don't know the internal details of how these
drives all work.

>  I would think this would be within seconds of
> successfully unmounting it.  Spinning for 30 minutes or more after it is
> unmounted sounds excessive to me, if it is only being spun by the firmware for
> flushing its journal buffers.

Indeed.  That really makes me wonder whether it is actually writing
anything.  It could just be that the drive has some vibration or
otherwise is giving the sensation that it is doing something when it
isn't.

Though, if the thing really does have a large journal inside to
improve performance it could actually buffer a pretty large number of
writes.  SMR drives can have a very large amount of write
amplification.  If each erase region on the disk contains 10k blocks
then every 1MB of data in the journal could potentially lead to 10GB
of disk rewrites, assuming that writes are randomly distributed.  It
also makes sense that when replaying the journal the drive is going to
prioritize erase regions with the most updates, which means that by
the time you're unmounting the drive you're going to expect the stuff
left in the journal to lead to the most write amplification.

I would imagine that a drive that works in this manner is going to use
a logfs for the journal (if the journal is even in SMR format), and
then it would just keep an erase region free at all times.  Anytime a
partial write happens in a region the data would get written to the
free region, then that region would be remapped in place of the old
region, and now the old region is unallocated for the next write.
This could be interrupted due to power loss at any time without any
real loss to data, since data is not overwritten in place.  Before the
journal record is closed out the old region is still valid, and after
the new region is valid.  An interrupted write is just repeated on
power up since the new region can just be overwritten again safely.

In any case, the bottom line is that a drive should be safe to unplug
if all filesystems on it are umounted and the umount commands all
return.  If the disk loses data after that point it is a manufacturing
design flaw.

-- 
Rich

Re: [gentoo-user] External hard drive and idle activity

[Mick]

On Thursday, 2 January 2020 14:43:58 GMT Rich Freeman wrote:


> Out of curiosity, what model drive is it?  Is it by chance an SMR /
> archive drive?  

Good catch!  I hadn't thought of this - the Linux kernel will need to have 
DM_ZONED enabled I think, for the OS to manager the shingled writes 
sequentially, but I don't have this enabled here because AFAIK I have no such 
drives in my possession. 


> Due to the limitations on how those write data out I
> could see them implementing an internal filesystem that journals
> incoming data and then writes it back out after the fact.

SMR drives which implement a 'device managed' write mechanism, will use their 
own firmware to control data storage.  The OS would not be aware of anything 
being different to a conventional drive.


> If so then
> that might happen even after the kernel thinks it is unmounted.
> However, such a drive firmware would probably use a journal that
> ensures data is safe even if power is cut mid-operation.  The drive
> isn't supposed to report that a write is completed until it is
> durable.

Which I take it to mean the drive would not be unmounted by the OS until it is 
safe to do so and for all intends and purposes it will also be safe to be 
powered down thereafter.  I would think this would be within seconds of 
successfully unmounting it.  Spinning for 30 minutes or more after it is 
unmounted sounds excessive to me, if it is only being spun by the firmware for 
flushing its journal buffers.  I have a conventional USB drive (WD passport) 
which is always spinning whether it is being written to or not.
-- 
Regards,

Mick

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] External hard drive and idle activity

[Rich Freeman]

On Thu, Jan 2, 2020 at 5:49 AM Dale  wrote:
>
> lol  I didn't think of that and I don't recall anyone else thinking of
> it either.

That is because syncing before unmounting doesn't do anything.  Unless
you use --lazy umount blocks until all writes are complete to a
device.  The instant it returns as far as the kernel is concerned the
device should be safe to power off.

If you do a sync first then of course the umount will complete more
quickly, since all writes should already be flushed.

I have no idea what your device is doing after it is unmounted, but it
doesn't have anything to do with the linux kernel unless some process
is directly accessing the raw device (very unlikely).  Maybe the drive
firmware is doing some kind of housekeeping, or maybe the drive has
some kind of vibration in it that just makes it feel like it is doing
something.  Or maybe the NSA or Red Army has hacked your firmware and
it is doing who knows what (yes, the NSA bit at least is a thing).  In
any case, chances are the drive manufacturer has accounted for sudden
power loss in the design because if they didn't there would be a ton
of complaints, since there is nothing you can do about this sort of
thing assuming the firmware is up to something.

Out of curiosity, what model drive is it?  Is it by chance an SMR /
archive drive?  Due to the limitations on how those write data out I
could see them implementing an internal filesystem that journals
incoming data and then writes it back out after the fact.  If so then
that might happen even after the kernel thinks it is unmounted.
However, such a drive firmware would probably use a journal that
ensures data is safe even if power is cut mid-operation.  The drive
isn't supposed to report that a write is completed until it is
durable.

-- 
Rich

Re: [gentoo-user] External hard drive and idle activity

[Francisco Ares]

Em qui., 2 de jan. de 2020 às 07:49, Dale  escreveu:

> Neil Bothwick wrote:
> > On Wed, 1 Jan 2020 20:27:34 -0600, Dale wrote:
> >
> >>> If you wait for a few seconds after the backup is completed before
> >>> you unmount the drive, you should be OK.  Although it may slow down
> >>> or any LEDs flash less frequently the drive may not stop spinning,
> >>> unless there is some power save process taking control of it.
> >>>
> >> Given the speed, it is likely done when I tell the KDE thingy to
> >> unmount.  Usually, I start the backup and walk away for a few minutes.
> >> I do it with one of my scripts, if one can call what I do a script, and
> >> it does the date command at the end.
> > Add a sync command to the end of the script to make sure all filesystem
> > buffers are flushed to disk before it finishes.
> >
> >
>
> Now there's a idea.  Between the sync command and the unmount process,
> it should be safe for sure.  You made good use of those brain cells.
> lol  I didn't think of that and I don't recall anyone else thinking of
> it either.  Still wonder what the heck it is doing sometimes tho.
>
> Dale
>
> :-)  :-)
>
>
Another thing to be considered is S.M.A.R.T. activity, that is proper to
the hard drive firmware.  Don't know if it is safe or not to power it down
during that process though.  I would wait until it really finishes all
activity.

But do you really need to power it down?  Mechanic (magnetic) hard drives
are known to have shorter lives if they are turned on and off frequently.
My desk computer is rarely turned off, I have 4 hard drives, the oldest is
close to 10 years old and SMART diagnostics are always good.

Best regards,
Francisco

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Neil Bothwick wrote:
> On Wed, 1 Jan 2020 20:27:34 -0600, Dale wrote:
>
>>> If you wait for a few seconds after the backup is completed before
>>> you unmount the drive, you should be OK.  Although it may slow down
>>> or any LEDs flash less frequently the drive may not stop spinning,
>>> unless there is some power save process taking control of it.
>>>  
>> Given the speed, it is likely done when I tell the KDE thingy to
>> unmount.  Usually, I start the backup and walk away for a few minutes. 
>> I do it with one of my scripts, if one can call what I do a script, and
>> it does the date command at the end.
> Add a sync command to the end of the script to make sure all filesystem
> buffers are flushed to disk before it finishes.
>
>

Now there's a idea.  Between the sync command and the unmount process,
it should be safe for sure.  You made good use of those brain cells. 
lol  I didn't think of that and I don't recall anyone else thinking of
it either.  Still wonder what the heck it is doing sometimes tho. 

Dale

:-)  :-) 

Re: [gentoo-user] External hard drive and idle activity

[Neil Bothwick]

On Wed, 1 Jan 2020 20:27:34 -0600, Dale wrote:

> > If you wait for a few seconds after the backup is completed before
> > you unmount the drive, you should be OK.  Although it may slow down
> > or any LEDs flash less frequently the drive may not stop spinning,
> > unless there is some power save process taking control of it.
> >  
> 
> Given the speed, it is likely done when I tell the KDE thingy to
> unmount.  Usually, I start the backup and walk away for a few minutes. 
> I do it with one of my scripts, if one can call what I do a script, and
> it does the date command at the end.

Add a sync command to the end of the script to make sure all filesystem
buffers are flushed to disk before it finishes.


-- 
Neil Bothwick

We've all heard that a million monkeys banging on a million
typewriters will eventually reproduce the works of Shakespeare.
Now, thanks to the Internet, we know this is not true.


pgpSDEl86DfD5.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Bill Kenworthy wrote:
> On 2/1/20 10:27 am, Dale wrote:
>> Mick wrote:
>>> On Thursday, 2 January 2020 00:09:14 GMT Dale wrote:
 Howdy,

 As some may recall, I have a 8TB external SATA hard drive that I do
 back
 ups on.  Usually, I back up once a day, more often if needed. 
 Usually I
 turn the power on, mount it, do the back ups, unmount and turn the
 power
 back off.  Usually it is powered up for 5 minutes or so.  When I
 unmount
 it tho, I sometimes notice it is still doing something.  I can feel
 the
 mechanism for the heads moving.  It has a slight vibration to it.
 Questions are, what is it doing and should I let it finish before
 powering it off?  I'd assume that once it in unmounted, the copy
 process
 is done so the files are safe.  I guess it is doing some sort of
 internal checks or something but I'm not sure.
>>> There is some delay with data still in the buffers between
>>> rsync/cp/tar/what-
>>> ever saying it's finished on your terminal and the drive itself
>>> finishing
>>> storing the data on the platters.
>>>
>>> If you look at vmstat, or keep an eye on Gkrelm you'll see what I mean.
>>> Normally, if you try to unmount a drive while it is still being
>>> written to,
>>> the umount/udisks command will complain the drive is busy.
>>>
>> When it does it for a somewhat short period of time, I can understand
>> that.  It's one reason I try to leave it on when it "feels" that it is
>> still busy.  Thing is, there are times when it goes on for 30 minutes or
>> more.  At those times, even a USB stick should be done.  One would think
>> at least.  It makes me curious as to what it is doing in that case.
>> Still, I'd rather the unmount command force a wait until it is done.
>> Honestly, I wouldn't want a drive or software that says something is
>> done when it isn't.  It's not good even when shutting a system down.
>> Given the speed of drives, I would think a few seconds at most.  Best to
>> be safe.  ;-)  I just wonder, is it doing two different things?  One
>> when it is busy for short periods of time and something else when it
>> goes on for a while.  This is what sort of puzzles me.  Selftest maybe??
>>
 Is it safe to turn it off even tho it is doing whatever it is doing?
 Should I wait?  Does it matter?

 Thanks.

 Dale

 :-)  :-)
>>> If you wait for a few seconds after the backup is completed before
>>> you unmount
>>> the drive, you should be OK.  Although it may slow down or any LEDs
>>> flash less
>>> frequently the drive may not stop spinning, unless there is some
>>> power save
>>> process taking control of it.
>>>
>> Given the speed, it is likely done when I tell the KDE thingy to
>> unmount.  Usually, I start the backup and walk away for a few minutes.
>> I do it with one of my scripts, if one can call what I do a script, and
>> it does the date command at the end.  Even if there was a lot of
>> changes, I can tell how long it was completed.  I try to give it a
>> couple minutes.  Still, good point.  This is one reason I'm asking about
>> this.  It's hard to know exactly what is going on here.
>>
 P. S. Down to last router that was discussed in another thread so I
 bought it while they had it.  Price may go up if I didn't.  Did more
 research on old modem, it is risky to try to convert to AT&T.  Some
 say
 not possible.
>>> Right, ISP controlled firmware typically requires re-flashing the
>>> device with
>>> the new ISP's firmware version.  In some cases even the boot code needs
>>> replacing.  Should you flash the router with a wrong firmware build,
>>> you could
>>> sometimes derive a door stop without additional cost.  In this case
>>> you'll
>>> need a JTAG and access to its circuit board with an OEM
>>> boot/firmware version
>>> to recover it.  In most cases OEMs support lines will redirect you
>>> to your
>>> ISP, who run an overseas support line and will ask you to reboot your
>>> MSWindows PC ... O_o
>>>
>>> This is a reason I avoid these kind of routers as much as I can.
>>>
>> Keep in mind, two pieces of hardware.  Router for the first two
>> sentences and Modem for next two.  Tried to be short so . . . . Anyway,
>> router should be flashable with Openwrt.  It's a slightly older model.
>> New model may be ready for flashing in a year or two but not so much at
>> the moment so I went with the older model. The modem, I never could find
>> the firmware.  I found links to it but those links ended up being dead.
>> Even if I had it, it was unlikely to work.  Possible but I'd be
>> concerned about its stability and such even if it did take it. I have a
>> modem and router on the way.  I just didn't want to miss the deal on the
>> router.  They had several a couple weeks or so ago.  I got the last
>> one.  Waiting for their arrival.
>>
>> Dale
>>
>> :-)  :-)
>>
>> Oh, I may post and see if anyone needs a Frontier modem later.  Maybe
>> someone on here could 

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Bill Kenworthy wrote:
> On 2/1/20 10:27 am, Dale wrote:
>> Mick wrote:
>>> On Thursday, 2 January 2020 00:09:14 GMT Dale wrote:
 Howdy,

 As some may recall, I have a 8TB external SATA hard drive that I do
 back
 ups on.  Usually, I back up once a day, more often if needed. 
 Usually I
 turn the power on, mount it, do the back ups, unmount and turn the
 power
 back off.  Usually it is powered up for 5 minutes or so.  When I
 unmount
 it tho, I sometimes notice it is still doing something.  I can feel
 the
 mechanism for the heads moving.  It has a slight vibration to it.
 Questions are, what is it doing and should I let it finish before
 powering it off?  I'd assume that once it in unmounted, the copy
 process
 is done so the files are safe.  I guess it is doing some sort of
 internal checks or something but I'm not sure.
>>> There is some delay with data still in the buffers between
>>> rsync/cp/tar/what-
>>> ever saying it's finished on your terminal and the drive itself
>>> finishing
>>> storing the data on the platters.
>>>
>>> If you look at vmstat, or keep an eye on Gkrelm you'll see what I mean.
>>> Normally, if you try to unmount a drive while it is still being
>>> written to,
>>> the umount/udisks command will complain the drive is busy.
>>>
>> When it does it for a somewhat short period of time, I can understand
>> that.  It's one reason I try to leave it on when it "feels" that it is
>> still busy.  Thing is, there are times when it goes on for 30 minutes or
>> more.  At those times, even a USB stick should be done.  One would think
>> at least.  It makes me curious as to what it is doing in that case.
>> Still, I'd rather the unmount command force a wait until it is done.
>> Honestly, I wouldn't want a drive or software that says something is
>> done when it isn't.  It's not good even when shutting a system down.
>> Given the speed of drives, I would think a few seconds at most.  Best to
>> be safe.  ;-)  I just wonder, is it doing two different things?  One
>> when it is busy for short periods of time and something else when it
>> goes on for a while.  This is what sort of puzzles me.  Selftest maybe??
>>
 Is it safe to turn it off even tho it is doing whatever it is doing?
 Should I wait?  Does it matter?

 Thanks.

 Dale

 :-)  :-)
>>> If you wait for a few seconds after the backup is completed before
>>> you unmount
>>> the drive, you should be OK.  Although it may slow down or any LEDs
>>> flash less
>>> frequently the drive may not stop spinning, unless there is some
>>> power save
>>> process taking control of it.
>>>
>> Given the speed, it is likely done when I tell the KDE thingy to
>> unmount.  Usually, I start the backup and walk away for a few minutes.
>> I do it with one of my scripts, if one can call what I do a script, and
>> it does the date command at the end.  Even if there was a lot of
>> changes, I can tell how long it was completed.  I try to give it a
>> couple minutes.  Still, good point.  This is one reason I'm asking about
>> this.  It's hard to know exactly what is going on here.
>>
 P. S. Down to last router that was discussed in another thread so I
 bought it while they had it.  Price may go up if I didn't.  Did more
 research on old modem, it is risky to try to convert to AT&T.  Some
 say
 not possible.
>>> Right, ISP controlled firmware typically requires re-flashing the
>>> device with
>>> the new ISP's firmware version.  In some cases even the boot code needs
>>> replacing.  Should you flash the router with a wrong firmware build,
>>> you could
>>> sometimes derive a door stop without additional cost.  In this case
>>> you'll
>>> need a JTAG and access to its circuit board with an OEM
>>> boot/firmware version
>>> to recover it.  In most cases OEMs support lines will redirect you
>>> to your
>>> ISP, who run an overseas support line and will ask you to reboot your
>>> MSWindows PC ... O_o
>>>
>>> This is a reason I avoid these kind of routers as much as I can.
>>>
>> Keep in mind, two pieces of hardware.  Router for the first two
>> sentences and Modem for next two.  Tried to be short so . . . . Anyway,
>> router should be flashable with Openwrt.  It's a slightly older model.
>> New model may be ready for flashing in a year or two but not so much at
>> the moment so I went with the older model. The modem, I never could find
>> the firmware.  I found links to it but those links ended up being dead.
>> Even if I had it, it was unlikely to work.  Possible but I'd be
>> concerned about its stability and such even if it did take it. I have a
>> modem and router on the way.  I just didn't want to miss the deal on the
>> router.  They had several a couple weeks or so ago.  I got the last
>> one.  Waiting for their arrival.
>>
>> Dale
>>
>> :-)  :-)
>>
>> Oh, I may post and see if anyone needs a Frontier modem later.  Maybe
>> someone on here could 

Re: [gentoo-user] External hard drive and idle activity

[Bill Kenworthy]

On 2/1/20 10:27 am, Dale wrote:

Mick wrote:

On Thursday, 2 January 2020 00:09:14 GMT Dale wrote:

Howdy,

As some may recall, I have a 8TB external SATA hard drive that I do back
ups on.  Usually, I back up once a day, more often if needed.  Usually I
turn the power on, mount it, do the back ups, unmount and turn the power
back off.  Usually it is powered up for 5 minutes or so.  When I unmount
it tho, I sometimes notice it is still doing something.  I can feel the
mechanism for the heads moving.  It has a slight vibration to it.
Questions are, what is it doing and should I let it finish before
powering it off?  I'd assume that once it in unmounted, the copy process
is done so the files are safe.  I guess it is doing some sort of
internal checks or something but I'm not sure.

There is some delay with data still in the buffers between rsync/cp/tar/what-
ever saying it's finished on your terminal and the drive itself finishing
storing the data on the platters.

If you look at vmstat, or keep an eye on Gkrelm you'll see what I mean.
Normally, if you try to unmount a drive while it is still being written to,
the umount/udisks command will complain the drive is busy.


When it does it for a somewhat short period of time, I can understand
that.  It's one reason I try to leave it on when it "feels" that it is
still busy.  Thing is, there are times when it goes on for 30 minutes or
more.  At those times, even a USB stick should be done.  One would think
at least.  It makes me curious as to what it is doing in that case.
Still, I'd rather the unmount command force a wait until it is done.
Honestly, I wouldn't want a drive or software that says something is
done when it isn't.  It's not good even when shutting a system down.
Given the speed of drives, I would think a few seconds at most.  Best to
be safe.  ;-)  I just wonder, is it doing two different things?  One
when it is busy for short periods of time and something else when it
goes on for a while.  This is what sort of puzzles me.  Selftest maybe??


Is it safe to turn it off even tho it is doing whatever it is doing?
Should I wait?  Does it matter?

Thanks.

Dale

:-)  :-)

If you wait for a few seconds after the backup is completed before you unmount
the drive, you should be OK.  Although it may slow down or any LEDs flash less
frequently the drive may not stop spinning, unless there is some power save
process taking control of it.


Given the speed, it is likely done when I tell the KDE thingy to
unmount.  Usually, I start the backup and walk away for a few minutes.
I do it with one of my scripts, if one can call what I do a script, and
it does the date command at the end.  Even if there was a lot of
changes, I can tell how long it was completed.  I try to give it a
couple minutes.  Still, good point.  This is one reason I'm asking about
this.  It's hard to know exactly what is going on here.


P. S. Down to last router that was discussed in another thread so I
bought it while they had it.  Price may go up if I didn't.  Did more
research on old modem, it is risky to try to convert to AT&T.  Some say
not possible.

Right, ISP controlled firmware typically requires re-flashing the device with
the new ISP's firmware version.  In some cases even the boot code needs
replacing.  Should you flash the router with a wrong firmware build, you could
sometimes derive a door stop without additional cost.  In this case you'll
need a JTAG and access to its circuit board with an OEM boot/firmware version
to recover it.  In most cases OEMs support lines will redirect you to your
ISP, who run an overseas support line and will ask you to reboot your
MSWindows PC ... O_o

This is a reason I avoid these kind of routers as much as I can.


Keep in mind, two pieces of hardware.  Router for the first two
sentences and Modem for next two.  Tried to be short so . . . . Anyway,
router should be flashable with Openwrt.  It's a slightly older model.
New model may be ready for flashing in a year or two but not so much at
the moment so I went with the older model. The modem, I never could find
the firmware.  I found links to it but those links ended up being dead.
Even if I had it, it was unlikely to work.  Possible but I'd be
concerned about its stability and such even if it did take it. I have a
modem and router on the way.  I just didn't want to miss the deal on the
router.  They had several a couple weeks or so ago.  I got the last
one.  Waiting for their arrival.

Dale

:-)  :-)

Oh, I may post and see if anyone needs a Frontier modem later.  Maybe
someone on here could use a spare or just needs one period, moving or
something.  Modem is wireless with a router as well.  Nice modem I guess.



Try atop from sys-process/atop - it will show you how busy individual 
disks are (and a lot of other stats as well.)


You can issue a sync command to flush any disk buffers before unmounting 
(umounting should sync as well.).  The heads may keep moving because of 
the internal data manageme

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Mick wrote:
> On Thursday, 2 January 2020 00:09:14 GMT Dale wrote:
>> Howdy,
>>
>> As some may recall, I have a 8TB external SATA hard drive that I do back
>> ups on.  Usually, I back up once a day, more often if needed.  Usually I
>> turn the power on, mount it, do the back ups, unmount and turn the power
>> back off.  Usually it is powered up for 5 minutes or so.  When I unmount
>> it tho, I sometimes notice it is still doing something.  I can feel the
>> mechanism for the heads moving.  It has a slight vibration to it. 
>> Questions are, what is it doing and should I let it finish before
>> powering it off?  I'd assume that once it in unmounted, the copy process
>> is done so the files are safe.  I guess it is doing some sort of
>> internal checks or something but I'm not sure. 
> There is some delay with data still in the buffers between rsync/cp/tar/what-
> ever saying it's finished on your terminal and the drive itself finishing 
> storing the data on the platters.
>
> If you look at vmstat, or keep an eye on Gkrelm you'll see what I mean.  
> Normally, if you try to unmount a drive while it is still being written to, 
> the umount/udisks command will complain the drive is busy.
>

When it does it for a somewhat short period of time, I can understand
that.  It's one reason I try to leave it on when it "feels" that it is
still busy.  Thing is, there are times when it goes on for 30 minutes or
more.  At those times, even a USB stick should be done.  One would think
at least.  It makes me curious as to what it is doing in that case. 
Still, I'd rather the unmount command force a wait until it is done. 
Honestly, I wouldn't want a drive or software that says something is
done when it isn't.  It's not good even when shutting a system down. 
Given the speed of drives, I would think a few seconds at most.  Best to
be safe.  ;-)  I just wonder, is it doing two different things?  One
when it is busy for short periods of time and something else when it
goes on for a while.  This is what sort of puzzles me.  Selftest maybe??

>> Is it safe to turn it off even tho it is doing whatever it is doing? 
>> Should I wait?  Does it matter? 
>>
>> Thanks.
>>
>> Dale
>>
>> :-)  :-) 
> If you wait for a few seconds after the backup is completed before you 
> unmount 
> the drive, you should be OK.  Although it may slow down or any LEDs flash 
> less 
> frequently the drive may not stop spinning, unless there is some power save 
> process taking control of it.
>

Given the speed, it is likely done when I tell the KDE thingy to
unmount.  Usually, I start the backup and walk away for a few minutes. 
I do it with one of my scripts, if one can call what I do a script, and
it does the date command at the end.  Even if there was a lot of
changes, I can tell how long it was completed.  I try to give it a
couple minutes.  Still, good point.  This is one reason I'm asking about
this.  It's hard to know exactly what is going on here. 

>> P. S. Down to last router that was discussed in another thread so I
>> bought it while they had it.  Price may go up if I didn't.  Did more
>> research on old modem, it is risky to try to convert to AT&T.  Some say
>> not possible. 
> Right, ISP controlled firmware typically requires re-flashing the device with 
> the new ISP's firmware version.  In some cases even the boot code needs 
> replacing.  Should you flash the router with a wrong firmware build, you 
> could 
> sometimes derive a door stop without additional cost.  In this case you'll 
> need a JTAG and access to its circuit board with an OEM boot/firmware version 
> to recover it.  In most cases OEMs support lines will redirect you to your 
> ISP, who run an overseas support line and will ask you to reboot your 
> MSWindows PC ... O_o
>
> This is a reason I avoid these kind of routers as much as I can.
>

Keep in mind, two pieces of hardware.  Router for the first two
sentences and Modem for next two.  Tried to be short so . . . . Anyway,
router should be flashable with Openwrt.  It's a slightly older model. 
New model may be ready for flashing in a year or two but not so much at
the moment so I went with the older model. The modem, I never could find
the firmware.  I found links to it but those links ended up being dead. 
Even if I had it, it was unlikely to work.  Possible but I'd be
concerned about its stability and such even if it did take it. I have a
modem and router on the way.  I just didn't want to miss the deal on the
router.  They had several a couple weeks or so ago.  I got the last
one.  Waiting for their arrival. 

Dale

:-)  :-)

Oh, I may post and see if anyone needs a Frontier modem later.  Maybe
someone on here could use a spare or just needs one period, moving or
something.  Modem is wireless with a router as well.  Nice modem I guess. 

Re: [gentoo-user] External hard drive and idle activity

[Dale]

Grant Taylor wrote:
> On 1/1/20 5:09 PM, Dale wrote:
>> Howdy,
>
> Hi,
>
>> As some may recall, I have a 8TB external SATA hard drive that I do
>> back ups on.  Usually, I back up once a day, more often if needed.
>> Usually I turn the power on, mount it, do the back ups, unmount and
>> turn the power back off.  Usually it is powered up for 5 minutes or
>> so. When I unmount it tho, I sometimes notice it is still doing
>> something. I can feel the mechanism for the heads moving.  It has a
>> slight vibration to it.  Questions are, what is it doing and should I
>> let it finish before powering it off?  I'd assume that once it in
>> unmounted, the copy process is done so the files are safe.  I guess
>> it is doing some sort of internal checks or something but I'm not sure.
>
> There might be some activity for up to 30 seconds after umount
> finishes and returns to the command prompt.
>
> Note:  umount will normally block until buffers are flushed to disk.
>
>> Is it safe to turn it off even tho it is doing whatever it is doing?
>
> I wouldn't.
>
>> Should I wait?
>
> I would.
>
>> Does it matter?
>
> Maybe.
>
> Is the drive SATA connected or USB connected to the machine?
>
> In some ways it doesn't matter.  You can tell the kernel to eject the
> drive.  Once that finishes, there is no active remnants of the drive
> in kernel.  5–15 seconds after that and you should be quite safe to
> power the drive off.
>
> echo 1 > /sys/class/block/$DEVICENAME/device/delete
>
> That will cause the kernel to gracefully disconnect the drive.
>
>> Thanks.
>
> :-)
>
>
>

It is connected with a eSATA cable.  It has a USB connector as well but
I just don't trust USB as much as I do SATA.  Other USB enclosures had
"issues".  ;-) 

If I touch the enclosure and feel it doing something, I leave it on,
just in case.  I actually been wondering about this for a while. 
Sometimes it will stop after a couple minutes, sometimes it is still
doing its thing 30 minutes later.  In the case of the first, I was
concerned about files being cached etc.  Thing is, it *should* do that
before unmounting.  In the case of it going on for 30 minutes or more, I
was wondering if it was doing some sort of housekeeping, media tests or
something.  I just didn't know if it was important to wait or not.  On
occasion I would leave the drive on overnight or all day.  I figure it
would be able to at least do a quick self test and report errors if any. 

Thanks for the extra info.  It helps a bit. 

Dale

:-)  :-) 

Re: [gentoo-user] External hard drive and idle activity

[Mick]

On Thursday, 2 January 2020 00:09:14 GMT Dale wrote:
> Howdy,
> 
> As some may recall, I have a 8TB external SATA hard drive that I do back
> ups on.  Usually, I back up once a day, more often if needed.  Usually I
> turn the power on, mount it, do the back ups, unmount and turn the power
> back off.  Usually it is powered up for 5 minutes or so.  When I unmount
> it tho, I sometimes notice it is still doing something.  I can feel the
> mechanism for the heads moving.  It has a slight vibration to it. 
> Questions are, what is it doing and should I let it finish before
> powering it off?  I'd assume that once it in unmounted, the copy process
> is done so the files are safe.  I guess it is doing some sort of
> internal checks or something but I'm not sure. 

There is some delay with data still in the buffers between rsync/cp/tar/what-
ever saying it's finished on your terminal and the drive itself finishing 
storing the data on the platters.

If you look at vmstat, or keep an eye on Gkrelm you'll see what I mean.  
Normally, if you try to unmount a drive while it is still being written to, 
the umount/udisks command will complain the drive is busy.


> Is it safe to turn it off even tho it is doing whatever it is doing? 
> Should I wait?  Does it matter? 
> 
> Thanks.
> 
> Dale
> 
> :-)  :-) 

If you wait for a few seconds after the backup is completed before you unmount 
the drive, you should be OK.  Although it may slow down or any LEDs flash less 
frequently the drive may not stop spinning, unless there is some power save 
process taking control of it.


> P. S. Down to last router that was discussed in another thread so I
> bought it while they had it.  Price may go up if I didn't.  Did more
> research on old modem, it is risky to try to convert to AT&T.  Some say
> not possible. 

Right, ISP controlled firmware typically requires re-flashing the device with 
the new ISP's firmware version.  In some cases even the boot code needs 
replacing.  Should you flash the router with a wrong firmware build, you could 
sometimes derive a door stop without additional cost.  In this case you'll 
need a JTAG and access to its circuit board with an OEM boot/firmware version 
to recover it.  In most cases OEMs support lines will redirect you to your 
ISP, who run an overseas support line and will ask you to reboot your 
MSWindows PC ... O_o

This is a reason I avoid these kind of routers as much as I can.

-- 
Regards,
Mick

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] External hard drive and idle activity

[Grant Taylor]

On 1/1/20 5:09 PM, Dale wrote:

Howdy,


Hi,

As some may recall, I have a 8TB external SATA hard drive that I do 
back ups on.  Usually, I back up once a day, more often if needed. 
Usually I turn the power on, mount it, do the back ups, unmount and 
turn the power back off.  Usually it is powered up for 5 minutes or so. 
When I unmount it tho, I sometimes notice it is still doing something. 
I can feel the mechanism for the heads moving.  It has a slight 
vibration to it.  Questions are, what is it doing and should I let it 
finish before powering it off?  I'd assume that once it in unmounted, 
the copy process is done so the files are safe.  I guess it is doing 
some sort of internal checks or something but I'm not sure.


There might be some activity for up to 30 seconds after umount finishes 
and returns to the command prompt.


Note:  umount will normally block until buffers are flushed to disk.


Is it safe to turn it off even tho it is doing whatever it is doing?


I wouldn't.


Should I wait?


I would.


Does it matter?


Maybe.

Is the drive SATA connected or USB connected to the machine?

In some ways it doesn't matter.  You can tell the kernel to eject the 
drive.  Once that finishes, there is no active remnants of the drive in 
kernel.  5–15 seconds after that and you should be quite safe to power 
the drive off.


echo 1 > /sys/class/block/$DEVICENAME/device/delete

That will cause the kernel to gracefully disconnect the drive.


Thanks.


:-)



--
Grant. . . .
unix || die





--
Grant. . . .
unix || die