Re: [9fans] SMART: Silly Marketing Acronym, Rebuts Truth

[Bakul Shah]

On Feb 3, 2018, at 3:59 PM, Lyndon Nerenberg  wrote:

>> The interesting thing (for me) was that
>> the SMART data from the drive gave it an all clear right to the end. But
>> unlike the SSDs, there was plenty of behavioural warning to remind me to
>> have the backups up to date and a spare at the ready...
> 
> FWIW, of the three-four dozen or so drives I have actively SMART monitored 
> over the years, of the ones that failed, *not* *one* gave a SMART warning 
> before dying.
> 
> That includes a spinny disk in one of my Mac Minis.  Of anyone, I would 
> expect Apple to be in bed with their HD suppliers enough to have HD firmware 
> that reliably reports SMART errors (since the disk utilities do pay attemtion 
> to it).  I spent a month listening to that drive's heads slam back to the 
> home position as it tried to recalibrate itself, before eventually dying. To 
> the bitter end, SMART reported "a-ok boss!"

I have only monitored disks @ home and for them SMART does report
all sorts of things. For recalibration to have kicked in, you should seeing
lots of errors. It’s possible Apple s/w dumbs it down & presents just one
binary choice to the user but the underlying data should be there. It’s
usually host s/w that falls short. Given disks’ bathtub error curve, s/w
should show error rate as a graph (as a function of time) from the
beginning and recommend replacement as the rate starts climbing
again. Ideally it should even compare it with averaged error data
collected by the vendor. History and good visualization can be very
helpful.

[9fans] SMART: Silly Marketing Acronym, Rebuts Truth

[Lyndon Nerenberg]

The interesting thing (for me) was that
the SMART data from the drive gave it an all clear right to the end. But
unlike the SSDs, there was plenty of behavioural warning to remind me to
have the backups up to date and a spare at the ready...


FWIW, of the three-four dozen or so drives I have actively SMART monitored 
over the years, of the ones that failed, *not* *one* gave a SMART warning 
before dying.


That includes a spinny disk in one of my Mac Minis.  Of anyone, I would 
expect Apple to be in bed with their HD suppliers enough to have HD 
firmware that reliably reports SMART errors (since the disk utilities do 
pay attemtion to it).  I spent a month listening to that drive's heads 
slam back to the home position as it tried to recalibrate itself, before 
eventually dying. To the bitter end, SMART reported "a-ok boss!"


--lyndon

Re: [9fans] Is fossil/venti file system a good choice for SSD?

[Bakul Shah]

On Sat, 03 Feb 2018 21:46:42 + "Digby R.S. Tarvin"  
wrote:
Digby R.S. Tarvin writes:
> 
>  Thats why I described my use case - to make the MTBF figures meaningful.  As
> I said, I have my system configured so that most heavy write accesses go to
> rotating media. I typically try to have my system partitions mounted read
> only, except var and tmp. I am currently using 32GB uSD devices for
> raspberry pi based servers, with about 100GB per year in writes (as
> reported by iostat), plus perhaps an initial 100GB in writing that occurs
> during installation and configuring. The last failure I had was about 3
> months ago, on uSD card that had been in use for just under 2 years.
>
> I can only speak of the experience that I have had, but I don't think my
> usage is sufficiently atypical to not count as 'in practice' for use as
> computer storage (ie I am not talking about  music players and cameras,
> phones etc).

Note that consumer grade uSD cards are much worse than SSDs --
they are mainly for the video recording market where capacity
is more important.  Not to mention The RasPis are poor at
reliability.  Even a xenon flash or near a RasPi could power a
RasPi2 down! And since they do no onboard power regulation,
people had lots of problems early on -- add one more USB
device and the thing can become unreliable.

At 100GB per year the SSD I am talking about should
last 3000 years!

Newer *high endurance* SanDisk microSDXC cards for example can
do 10k hours of video recording at a rate of 26Mbps speed.
That translates to about 117TBW. At 100GB writes/year they
should last over 1000 years.

> So from my experience, I would still tend to go along with Erik's advice
> (as relayed by Steve), or perhaps be even more fastidious about backups
> when using flash...

I think this has shifted more in favor of SSDs in the past
couple years.

The convention wisdom for ZFS was to not use SSDs for its
intent log (as all writes go through it) but it was okay to
use SSDs for the read cache, L2ARC (though that may not be
very effective). This is similar to Erik's recommendation.
But this falls apart if you use SSDs as your main storage.

To use a 300TBW device for 10 years, you'd have to *average*
over 82GB a day per device. I certainly don't see traffic
anywhere near that. If it weren't for the cost, I would
rebuilt my zfs array using SSDs. [Note that ZFS does its own
checksumming and has redundancy of its own]

Re: [9fans] Is fossil/venti file system a good choice for SSD?

[Digby R.S. Tarvin]

 Thats why I described my use case - to make the MTBF figures meaningful.  As
I said, I have my system configured so that most heavy write accesses go to
rotating media. I typically try to have my system partitions mounted read
only, except var and tmp. I am currently using 32GB uSD devices for
raspberry pi based servers, with about 100GB per year in writes (as
reported by iostat), plus perhaps an initial 100GB in writing that occurs
during installation and configuring. The last failure I had was about 3
months ago, on uSD card that had been in use for just under 2 years.

I can only speak of the experience that I have had, but I don't think my
usage is sufficiently atypical to not count as 'in practice' for use as
computer storage (ie I am not talking about  music players and cameras,
phones etc).

The symptom tends to be thousands of widely dispersed bad sectors appearing
almost simulataneously, in this case on a journaling filesystem, so
previously valid information goes bad without any access being made to it.
Perhaps this is wear leveling going wrong when it is trying to move less
frequently used data to parts of the flash which are very worn. Whatever it
is, it seems subjectively to be a much more rapid decline than rotating
media when it starts going wrong, and whereas rotating media usually
returns sporadic read errors when failing, I have found SSD's often
silently return the wrong data when they go bad - which I find particularly
worrying (you wouldn't want to be using a SSD in a RAID, for example).
Consequently I have started using filesystems which checksum data as well
as metadata when using flash based storage.

My usage of 2.5" SATA SSD's has not really been over a long enough period
to get a good feel for how it compares with removable flash media - I would
hope it is more robust. But when used as the only storage in a laptop
environment, I would expect much higher levels or write access than a
specially configured server. I don't pretend to know what 'typical things
people do' would be, but it isn't hard to imagine scenarios that could
result in several GB's a day for a non-technical user - downloading movies
to a laptop to watch while commuting for example... I certainly wouldn't
feel comfortable regularly rebuilding the Linux kernel on a SSD based
machine.

So from my experience, I would still tend to go along with Erik's advice
(as relayed by Steve), or perhaps be even more fastidious about backups
when using flash...



On 3 February 2018 at 20:10, Bakul Shah  wrote:

> On Sat, 03 Feb 2018 18:49:50 + "Digby R.S. Tarvin" 
> wrote:
> Digby R.S. Tarvin writes:
> >
> > My experience of running normal (read mostly) Linux filesystems on solid
> > state media is that SSD is more robust but far less reliable than
> rotating
> > media.
> >
> > MTBF for rotating media for me has been around 10 years. MTBF for SSD has
> > been about 2. And the SSD always seems to fail catastrophically -
> appearing
> > to work fine one day, then after an inexplicable crash, half the media is
> > unreadable. I assume this is something to do with the wear leveling,
> which
> > successfully gets most of the blocks to wear out at the same time with no
> > warning. If I reformat and reload the SSD to correct all the mysterious
> > corruptions, it last a few weeks, then does the same thing again.
>
> MTTF doesn't make much sense for SSDs. A 1TB SSD I bought a
> couple years ago has a rating of 300 TB written, an MTBF of 2M
> hours and a 10 year warranty. It can do over 500MB/s of
> sequential writes. If I average 9.5MB/s writes, it will last
> year. If I continuoulsy write 100MB/s, it will last under 35
> days. In contrast the life of an HDD depends on how long it
> has been spinning, seeks, temperature and load/unloads. A disk
> with 5 year warraty will likely last 5 years even if you write
> 100MB/s continuously.
>
> And consumer HDSC cards such as the ones used in cameras and
> Raspi are much much worse.
>
> In practice an SSD will last much longer than a HDD since
> average write rates are not high for the typical things people
> do.
>
>

Re: [9fans] Is fossil/venti file system a good choice for SSD?

[Bakul Shah]

On Sat, 03 Feb 2018 18:49:50 + "Digby R.S. Tarvin"  
wrote:
Digby R.S. Tarvin writes:
> 
> My experience of running normal (read mostly) Linux filesystems on solid
> state media is that SSD is more robust but far less reliable than rotating
> media.
> 
> MTBF for rotating media for me has been around 10 years. MTBF for SSD has
> been about 2. And the SSD always seems to fail catastrophically - appearing
> to work fine one day, then after an inexplicable crash, half the media is
> unreadable. I assume this is something to do with the wear leveling, which
> successfully gets most of the blocks to wear out at the same time with no
> warning. If I reformat and reload the SSD to correct all the mysterious
> corruptions, it last a few weeks, then does the same thing again.

MTTF doesn't make much sense for SSDs. A 1TB SSD I bought a
couple years ago has a rating of 300 TB written, an MTBF of 2M
hours and a 10 year warranty. It can do over 500MB/s of
sequential writes. If I average 9.5MB/s writes, it will last
year. If I continuoulsy write 100MB/s, it will last under 35
days. In contrast the life of an HDD depends on how long it
has been spinning, seeks, temperature and load/unloads. A disk
with 5 year warraty will likely last 5 years even if you write
100MB/s continuously.

And consumer HDSC cards such as the ones used in cameras and
Raspi are much much worse.

In practice an SSD will last much longer than a HDD since
average write rates are not high for the typical things people
do.

Re: [9fans] Is fossil/venti file system a good choice for SSD?

[Digby R.S. Tarvin]

My experience of running normal (read mostly) Linux filesystems on solid
state media is that SSD is more robust but far less reliable than rotating
media.

MTBF for rotating media for me has been around 10 years. MTBF for SSD has
been about 2. And the SSD always seems to fail catastrophically - appearing
to work fine one day, then after an inexplicable crash, half the media is
unreadable. I assume this is something to do with the wear leveling, which
successfully gets most of the blocks to wear out at the same time with no
warning. If I reformat and reload the SSD to correct all the mysterious
corruptions, it last a few weeks, then does the same thing again.

I have had servers running of solid state media continuously since about
2003. Using PATA to CF adapters initially, currently uSD in raspberry pi
etc, and 2.5" SATA SSD drives. I used to use mostly SCSI rotating media, so
my reliability may have been better than cheaper PC drives. I had quite a
few (probably 90%) of the 1GB Wren 7 drives retired after 10-15 years of
running 24/7 under very heavy load (in an on-air broadcast system) with no
signs of trouble. The 2.5" SATA form factor SSD's seem to last better -
perhaps indicating that the advertised capacity is a smaller proportion of
overall capacity available to level the wear over..

I don't have a large number of servers, so not really a big enough sample
to draw definite conclusions from, but enough to make me wary of relying
too much on SSD's as a panacea.

My current server configuration is a uSD system drive, with a much larger
rotating disk that spins down when not in use (generally only gets used
when a user is logged in), and an up to date backup of everything on the
uSD is kept on the rotating media. I am not keen on having SSD as a swap
device, unless you have multiple SSD's, in which case you just treat the
swap/tmp media as disposable. If I am short of ram (like on a raspberry
pi), I would prefer to have an external ram drive for swapping.

I have had the rotating media fail once in this configuration - quite
recently. 1TB 5.25", so quite a few years old.  It went through a couple of
months of taking too long to spin up when accessed after a spin down,
requiring a manual unmount to get the system to recognize it again. Then
eventually wouldn't spin up at all. The interesting thing (for me) was that
the SMART data from the drive gave it an all clear right to the end. But
unlike the SSDs, there was plenty of behavioural warning to remind me to
have the backups up to date and a spare at the ready...

So bottom line, in my experience, SSDs are great for read access time, low
power, low noise, and robustness. But they are not good for for
reliability, capacity or usage which is not read-mostly. (and RAID usage is
no substitute for backups - but that is another story)

DigbyT.

On 3 February 2018 at 16:53, hiro <23h...@gmail.com> wrote:

> not so sure about mtbf. but it's too early to tell.
>
>

Re: [9fans] Is fossil/venti file system a good choice for SSD?

[hiro]

not so sure about mtbf. but it's too early to tell.

Re: [9fans] Is fossil/venti file system a good choice for SSD?

[Steve Simon]

More the lidea of reading from the ssd first.
The way fs(4) driver works is you order your drives
with the first written at one end of the queue and first read at the other.

I assume the read rate and rotational latency of an ssd should
be better than hdd so I get the best performace.

I should add that Erik once warned me (and I believe him) that ssds are
really no more reliable in terms of MTBF than hhds, its just that they are more
robust to impact - just in case anyone thinks they last forever...

-Steve

Re: [9fans] Is fossil/venti file system a good choice for SSD?

[hiro]

what's the rationale behind writing to the hdd first?

Re: [9fans] Is fossil/venti file system a good choice for SSD?

[Steve Simon]

i would say ssd is an excellent chiice for venti, the argument is less clear 
for fossil which us much more like a traditional filesystem.

fossil and venti do not have the performance if a modern filesystem bur an ssd 
can make them fast enough for most use (i dont stream movies from my plan9 
system).

personally i have three disks mirrored, two rotating disks and, last year i 
added an sdd, they are irdered in the mirror so the ssd is read first but the 
winchesters are written first.

-Steve

> On 3 Feb 2018, at 09:39, lchg  wrote:
> 
> As I know,  fossil/venti file system is log-structured,  so it may be good 
> for flash devices, especially in extending life of flash devices.

Re: [9fans] Is fossil/venti file system a good choice for SSD?

[Ethan Grammatikidis]

On Sat, Feb 3, 2018, at 9:39 AM, lchg wrote:
> As I know,  fossil/venti file system is log-structured,  so it may be
> good for flash devices, especially in extending life of flash devices.
As far as I know the device itself will even the wear by remapping the
blocks it presents to the computer. I remember documentation for Linux
jffs saying there's no point using it on removable flash storage because
it does that, it's only raw flash memory ICs which need filesystem
support. I'm sure SSDs do their own wear leveling too, because, like
removable flash storage, they're sold for use with fat32, ntfs, etc.
--
The lyf so short, the craft so long to lerne. -- Chaucer

[9fans] Is fossil/venti file system a good choice for SSD?

[lchg]

As I know,  fossil/venti file system is log-structured,  so it may be good
for flash devices, especially in extending life of flash devices.