Re: Help! secure boot is preventing boot of debian

[Richmond]

"Thomas Schmitt"  writes:

> Hi,
>
> Richmond wrote:
>> OK I got it booted and re-installed grub from debian. But I don't
>> know why it happened, I haven't changed any keys or done anything
>> except an opensuse update. I will ask the opensuse list
>
> I remember to have seen discussions about newly installed shim adding
> names of older shims or bootloaders to something called SBAT.  I find
> in my mailbox a mail with a link to
> https://bugzilla.opensuse.org/show_bug.cgi?id=1209985
>
> About SBAT i found in the web:
>   
> https://www.gnu.org/software/grub/manual/grub/html_node/Secure-Boot-Advanced-Targeting.html
>   https://github.com/rhboot/shim/blob/main/SBAT.md
>
>
> Have a nice day :)
>
> Thomas

Thanks. They have a wiki on how to fix this:

https://en.opensuse.org/openSUSE:UEFI#Reset_SBAT_string_for_booting_to_old_shim_in_old_Leap_image

I found re-installing debian's grub easier, until next time perhaps...

Re: Help! secure boot is preventing boot of debian

[Thomas Schmitt]

Hi,

Richmond wrote:
> OK I got it booted and re-installed grub from debian. But I don't know
> why it happened, I haven't changed any keys or done anything except an
> opensuse update. I will ask the opensuse list

I remember to have seen discussions about newly installed shim adding
names of older shims or bootloaders to something called SBAT.
I find in my mailbox a mail with a link to
  https://bugzilla.opensuse.org/show_bug.cgi?id=1209985

About SBAT i found in the web:
  
https://www.gnu.org/software/grub/manual/grub/html_node/Secure-Boot-Advanced-Targeting.html
  https://github.com/rhboot/shim/blob/main/SBAT.md


Have a nice day :)

Thomas

Re: Help! secure boot is preventing boot of debian

[Richmond]

Marco Moock  writes:

> Am 01.06.2024 um 20:01:43 Uhr schrieb Richmond:
>
>> Should I disable secure boot temporarily? will that allow booting?
>
> That should allow booting it.
>
> Have you changed anything at the keys in the EFI (maybe UEFI
> firmware update)?

OK I got it booted and re-installed grub from debian. But I don't know
why it happened, I haven't changed any keys or done anything except an
opensuse update. I will ask the opensuse list

Re: Help! secure boot is preventing boot of debian

[Marco Moock]

Am 01.06.2024 um 20:01:43 Uhr schrieb Richmond:

> Should I disable secure boot temporarily? will that allow booting?

That should allow booting it.

Have you changed anything at the keys in the EFI (maybe UEFI
firmware update)?

-- 
Gruß
Marco

Send unsolicited bulk mail to 1717264903mu...@cartoonies.org

Help! secure boot is preventing boot of debian

[Richmond]

I have a PC with two operating systems installed, Debian, and Opensuse.
Both are installed with Secure Boot. Each has its own grub installation.
Normally I boot debian, and if I want to boot opensuse I select UEFI
settings from the main menu and select opensuse from there which
launches the opensuse grub. Today I booted opensuse, and did an update
which included an update to grub. Now I cannot boot debian as it says
bad shim or bad signature.

Each grub menu has the alternate O.S. on it, but booting debian from the
opensuse grub menu did not work either.

Should I disable secure boot temporarily? will that allow booting?

RE: Shopify help center

[Mublex Kion]

Hello store owner, how are you doing today, I am Mublex Kion, a shopify
expert, I visited your store recently and I appreciate your effort towards
setting up the store, However towards my analysis I can see that you have
not implemented the latest strategy used by successful Shopify store owners
in the recent month which is Google Shopping Ads(GSAs). If I should ask,
Did you know what GSAs is and how it can benefit your store?

Re: Help to report a bug related to a usb3 lan adapter driver

[Charles Curley]

On Mon, 15 Apr 2024 20:57:00 +0200
user7415 same  wrote:

> I had a discussion in stack exchange related to the problem that is
> well explained here:
> https://unix.stackexchange.com/questions/774594/debian-12-all-of-sudden-my-usb3-lan-adapter-get-assigned-random-mac-address-ea
> 
> For what I understood the problem was fixed in 6.8, but I'm using
> debian 12 that will never use that so much new kernel I guess, could
> you help me to report officially the bug so that the upstream channel
> will correct it by the 6.1.0-22 version ?

Bookwom backports has linux-image-6.6.13+bpo-amd64. You might try that.
https://backports.debian.org/

It just so happens I have one of the same beasties. I just plugged it
in to a machine running kernel 6.6.13+bpo-amd64, unplugged it, waited
20 seconds, and plugged it in to another machine running kernel
6.5.0-0.deb12.4-amd64. I then plugged it into a machine with
6.6.13+bpo-amd64. All three times I got a MAC address of
8c:ae:4c:d6:22:17. So either of those kernels might well work for you.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Help to report a bug related to a usb3 lan adapter driver

[user7415 same]

Hello,

I'm not very skilled in what concerns to kernel patching and compiling.

I had a discussion in stack exchange related to the problem that is well
explained here:
https://unix.stackexchange.com/questions/774594/debian-12-all-of-sudden-my-usb3-lan-adapter-get-assigned-random-mac-address-ea

For what I understood the problem was fixed in 6.8, but I'm using debian 12
that will never use that so much new kernel I guess, could you help me to
report officially the bug so that the upstream channel will correct it by
the 6.1.0-22 version ?

Thank you very much!

Re: help needed to get a bookworm install to succeed

[Curt]

On 2024-04-01, Michel Verdier  wrote:
> On 2024-04-01, DdB wrote:
>
>>> A computer with a 6-core processor, 64 GB memory, and 9 drive bays/
>>> ports that cannot boot USB?  That does not make sense.
>>
>> Why not?
>
> Perhaps because usb boot is available since a very long time
>

The OP informed us that the board was over ten years old, and does not
offer USB booting.

I would assume he would know, and you would not.

SOLVED (was: Re: help needed to get a bookworm install to succeed)

[DdB]

Am 01.04.2024 um 18:52 schrieb David Christensen:
> A bad USB flash drive would explain why you cannot boot the Debian
> installer.  Please buy a good quality USB 3.0+ flash drive and try again.

A friend of mine just let me use an external CD-Drive with the netboot
image. This is already the third time, i am restarting the installation
process, due to my false assumptions about the intelligence within the
installer.

The last time, i was quite happy until i came to notice, that partitions
were not aligned with physical sector boundaries, which i assumed would
be elementary best practice.

But apart from losing some of my illusions the hard way, all is well.
A big thank you to all the crowd offering suggestions and encouragement.

so long, DdB

Re: help needed to get a bookworm install to succeed

[David Christensen]

On 4/1/24 03:10, DdB wrote:

Am 01.04.2024 um 07:44 schrieb David Christensen:

Please post a console session that identifies the ISO you are using,
verifies the checksum, burns the ISO to a USB flash drive, and compares
the ISO against the flash drive.


Ok, in the meantime, i came to similar conclusions and found that the
USB-stick i was using, had consistent read errors at the first 2
gigabytes after having been used for years as memory extension in my
router. Fixed that and will replace the stick.



A bad USB flash drive would explain why you cannot boot the Debian 
installer.  Please buy a good quality USB 3.0+ flash drive and try again.



David

Re: help needed to get a bookworm install to succeed

[Michel Verdier]

On 2024-04-01, DdB wrote:

>> A computer with a 6-core processor, 64 GB memory, and 9 drive bays/
>> ports that cannot boot USB?  That does not make sense.
>
> Why not?

Perhaps because usb boot is available since a very long time

> *should* is the correct word. The board being over 10 years old, it does
> not offer USB booting, no way.

I have one 20+ old which can usb boot but need to switch it in the
bios. The usb choice appears in the bios only after having plugged the
usb device. And of course detecting a valid usb device. You should check
that.

Re: help needed to get a bookworm install to succeed

[DdB]

Am 01.04.2024 um 07:44 schrieb David Christensen:
> 
> 
> A computer with a 6-core processor, 64 GB memory, and 9 drive bays/
> ports that cannot boot USB?  That does not make sense.

Why not?

> 
> 
> Please post a console session that identifies the ISO you are using,
> verifies the checksum, burns the ISO to a USB flash drive, and compares
> the ISO against the flash drive.

Ok, in the meantime, i came to similar conclusions and found that the
USB-stick i was using, had consistent read errors at the first 2
gigabytes after having been used for years as memory extension in my
router. Fixed that and will replace the stick.

> 
> 
> Then insert the USB flash drive into a USB port on the the target
> computer, power up and enter Setup, reset the settings to factory
> defaults, enable USB booting, set the USB flash drive as the first boot
> device, save, and exit.  The Debian installer should then boot.

*should* is the correct word. The board being over 10 years old, it does
not offer USB booting, no way. It is an early server board that supports
that much ECC, which is great for zfs.
> 
> 
> David

But i received many hints and ideas and just have to wait for a friend
of mine to overcome my physical handicap to see some progress. :-)

Tx 2 everyone
DdB

Re: help needed to get a bookworm install to succeed

[David Christensen]

On 3/31/24 02:18, DdB wrote:

Hello list,

i intend to create a huge backup server from some oldish hardware.
Hardware has been partly refurbished and offers 1 SSD + 8 HDD on a 6core
Intel with 64 GB RAM.
Already before assembling the hardware, grub was working from the SSD,
which got lvm partitioning and is basically empty. As i have no working
CD drive nor can this old machine boot from USB, i put an ISO for
bookworm onto an lvm-LV. Using grub, i can manually boot from that ISO
and see the first installer screens. But after asking some questions,
the installer wants to mount the external media (ISO), and does not find
it on sd[a-z], then aborts.
By switching to Desktop 4, i can see the attempt to search for the
"CD"-drive, which is bound to fail.
I am not familiar with the very restricted shell, that is available from
the installer (busybox) and have not yet found an approach to circumvent
my problems. i would like to use the installer, as debootstrapping would
necessitate alot more knowledge than mine.

Suggestions are welcome :-)
DdB




A computer with a 6-core processor, 64 GB memory, and 9 drive bays/ 
ports that cannot boot USB?  That does not make sense.



Please post a console session that identifies the ISO you are using, 
verifies the checksum, burns the ISO to a USB flash drive, and compares 
the ISO against the flash drive.



Then insert the USB flash drive into a USB port on the the target 
computer, power up and enter Setup, reset the settings to factory 
defaults, enable USB booting, set the USB flash drive as the first boot 
device, save, and exit.  The Debian installer should then boot.



David

Re: help needed to get a bookworm install to succeed

[David Wright]

On Sun 31 Mar 2024 at 11:18:30 (+0200), DdB wrote:

> Already before assembling the hardware, grub was working from the SSD,
> which got lvm partitioning and is basically empty. As i have no working
> CD drive nor can this old machine boot from USB, i put an ISO for
> bookworm onto an lvm-LV. Using grub, i can manually boot from that ISO
> and see the first installer screens. But after asking some questions,
> the installer wants to mount the external media (ISO), and does not find
> it on sd[a-z], then aborts.
> By switching to Desktop 4, i can see the attempt to search for the
> "CD"-drive, which is bound to fail.
> I am not familiar with the very restricted shell, that is available from
> the installer (busybox) and have not yet found an approach to circumvent
> my problems. i would like to use the installer, as debootstrapping would
> necessitate alot more knowledge than mine.

My memory of doing this is rusty, as it's a while since my
Seattle2 machine finally expired. I would try downloading the
kernel¹ and initrd from:

  
http://http.us.debian.org/debian/dists/bookworm/main/installer-amd64/current/images/netboot/debian-installer/amd64/

as these can search for the ISO in a greater range of locations.
I'd copy the two files onto the hard disk, and use an entry like:

  menuentry "Install Debian via HTTP" {
search --no-floppy --label --set=root noah03
linux   /boot/linux priority=low
initrd  /boot/initrd.gz
  }

in Grub to boot it. (Add a custom entry, or just edit a preexisting
entry to suit. BTW I use LABELs on my disks.) Make sure the kernel
versions are the same for those two files and the ISO.

https://www.debian.org/releases/bookworm/amd64/apas02.en.html#howto-getting-images-hard-disk
https://www.debian.org/releases/bookworm/amd64/ch05s01.en.html#boot-initrd
https://www.debian.org/releases/bookworm/amd64/ch04s04.en.html

¹ I see linux, rather than vmlinuz, at that location now.

Cheers,
David.

Re: help needed to get a bookworm install to succeed

[Michael Kjörling]

On 31 Mar 2024 11:18 +0200, from debianl...@potentially-spam.de-bruyn.de (DdB):
> As i have no working
> CD drive nor can this old machine boot from USB, i put an ISO for
> bookworm onto an lvm-LV. Using grub, i can manually boot from that ISO
> and see the first installer screens. But after asking some questions,
> the installer wants to mount the external media (ISO), and does not find
> it on sd[a-z], then aborts.

I would suggest to write the _same_ ISO file to a USB stick of
sufficient size, and leave the USB stick connected while running the
installer. The installer should detect the USB stick and use that as
the source for installation, regardless of how you booted into the
installer.

As long as both media contain the same data, this should be completely
unproblematic.

Think of it as a variation of, in the old days, booting the installer
from a floppy (on a system that couldn't boot from CD) but actually
installing from a CD.

-- 
Michael Kjörling  https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Re: help needed to get a bookworm install to succeed

[Felix Miata]

DdB composed on 2024-03-31 11:18 (UTC+0200):

> Suggestions are welcome :-)

https://www.debian.org/CD/netinst/

All my installations use this NET method. What I usually do though is extract
linux and initrd.gz from it or directly from the mirrors and load them with Grub
rather than booting the NET .iso.
-- 
Evolution as taught in public schools is, like religion,
based on faith, not based on science.

 Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata

Re: help needed to get a bookworm install to succeed

[Geert Stappers]

On Sun, Mar 31, 2024 at 11:18:30AM +0200, DdB wrote:
> Hello list,
> 
> i intend to create a huge backup server from some oldish hardware.
> Hardware has been partly refurbished and offers 1 SSD + 8 HDD on a 6core
> Intel with 64 GB RAM.
> Already before assembling the hardware, grub was working from the SSD,
> which got lvm partitioning and is basically empty. As i have no working
> CD drive nor can this old machine boot from USB, i put an ISO for
> bookworm onto an lvm-LV.

Not knowing how that was done, I guess disk was taken to another
computer where the lvm-LV was written.

If so:  put the (ISO)image just on the disk, not in LVM.



> Using grub, i can manually boot from that ISO
> and see the first installer screens. But after asking some questions,
> the installer wants to mount the external media (ISO), and does not find
> it on sd[a-z], then aborts.
> By switching to Desktop 4, i can see the attempt to search for the
> "CD"-drive, which is bound to fail.
> I am not familiar with the very restricted shell, that is available from
> the installer (busybox) and have not yet found an approach to circumvent
> my problems. i would like to use the installer, as debootstrapping would
> necessitate alot more knowledge than mine.
> 
> Suggestions are welcome :-)

Original post based:
  Take bootdisk out the back server,
  take the disk to other server.
  Install there, move the disk to the back server.

What I would do:
  Network boot


> DdB
 

Groeten
Geert Stappers
-- 
Silence is hard to parse

help needed to get a bookworm install to succeed

[DdB]

Hello list,

i intend to create a huge backup server from some oldish hardware.
Hardware has been partly refurbished and offers 1 SSD + 8 HDD on a 6core
Intel with 64 GB RAM.
Already before assembling the hardware, grub was working from the SSD,
which got lvm partitioning and is basically empty. As i have no working
CD drive nor can this old machine boot from USB, i put an ISO for
bookworm onto an lvm-LV. Using grub, i can manually boot from that ISO
and see the first installer screens. But after asking some questions,
the installer wants to mount the external media (ISO), and does not find
it on sd[a-z], then aborts.
By switching to Desktop 4, i can see the attempt to search for the
"CD"-drive, which is bound to fail.
I am not familiar with the very restricted shell, that is available from
the installer (busybox) and have not yet found an approach to circumvent
my problems. i would like to use the installer, as debootstrapping would
necessitate alot more knowledge than mine.

Suggestions are welcome :-)
DdB

Re: Problem with sleeping mode ( debian 12 ) please help

[Brad Rogers]

On Tue, 5 Mar 2024 15:09:34 +0100
Mansour Nasri  wrote:

Hello Mansour,

>Hi I'm using debian 12 in Lenovo yoga legion core i5 12th-gen with
>Nvidia
{cut}

You asked this, or a very similar question, on 29 Feb.  You had two
responses that I saw.  I suggest you review those replies and respond
accordingly.

-- 
 Regards  _   "Valid sig separator is {dash}{dash}{space}"
 / )  "The blindingly obvious is never immediately apparent"
/ _)rad   "Is it only me that has a working delete key?"
This is the fifty first state of the USA
Heartland - The The


pgpj5DMEWWoAs.pgp
Description: OpenPGP digital signature

Problem with sleeping mode ( debian 12 ) please help

[Mansour Nasri]

Hi I'm using debian 12 in Lenovo yoga legion core i5 12th-gen with Nvidia
RTX 3050 and i'm facing a serious using debian 12 on this PC,
When the PC is on sleep mode ( suspend ) it's doesn't wake up anymore until
forcing shutting down and this each time the PC turns on suspend mode, (
fastboot are disabled )of course, the PC wake up but the screen is totally
black nothing displayed on the screen, ( installed Nvidia drivers from the
APT repo ) and is same problem.

"on my old PC dell i7 10th ( no additional GPU ) i never had this kind of
issue",  please help to resolve this problem I really don't want to back to
windows anymore. Thank you so much

Re: please, help to get the image write done, due to an error. Thank you!

[Steve McIntyre]

Moving this to the debian-user list and setting reply-to accordingly...

On Fri, Feb 09, 2024 at 12:25:15PM +, guido mezzalana wrote:
>Hello
>
>First of all I wish to thank you all Debian's Team! To still enjoy a free OS:)
>
>I am running Ubuntu XFCE and I am using the Disk Image Write to get your lates
>Debian 12.4.1 XFCE on my USB. Unfortunately after a few try I get always an
>error and so I cannot get the job done. I do have a Lenovo X200 which comes
>without DVD writer.

Ummm. What image exactly are you trying to write, and how?

We don't have any images labelled with version 12.4.1. Where did you
get this image from?

What exact errors is the image writer program reporting? Without that
information it's very difficult to help you.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
< sladen> I actually stayed in a hotel and arrived to find a post-it
  note stuck to the mini-bar saying "Paul: This fridge and
  fittings are the correct way around and do not need altering"

Re: smartctl cannot access my storage, need syntax help

[David Wright]

On Thu 25 Jan 2024 at 12:24:21 (-0500), Roy J. Tellason, Sr. wrote:
> On Thursday 25 January 2024 09:03:36 am Anssi Saari wrote:
> > On Tue 23 Jan 2024 at 06:32:54 (-0500), gene heskett wrote:
> > > On 1/23/24 06:12, Gremlin wrote:
> > > > On 1/23/24 06:04, gene heskett wrote:
> > > > > On 1/23/24 00:30, Karl Vogel wrote:
> > > > > > > > On 1/22/24 11:31, gene heskett wrote:
> > > > > > 
> > > > > > G> How does an 8T backup server sound for another $200 in hdwe?  
> > > > > > Very
> > > > > > G> enticing and I do have the sheckel's.
> > > > > > 
> > > > > > https://www.amazon.com/dp/B07CQJBSQL
> > > > > > Seagate Desktop 8TB external Hard Drive, 3.5 Inch, USB 3.0 
> > > > > > STGY8000400
> > > > > > $168.18
> > > > > > 
> > > > > > What if you buy two, use one for a complete backup and the other for
> > > > > > incrementals or differentials?  (I know, more than $200...)
> > > > > > 
> > > > > My disastrous experience with the last pair of seagates
> > > > > preclude exploring that path, ever again. I bought a couple
> > > > > 2T's to replace  2 1T's I had outgrown and had close to 70,000
> > > > > spining hours on the.  They lasted a bit less than 30 days,
> > > > > dropping off the sata cables connecting then, never to be
> > > > > found again. Then I find they were shingled tech, and helium
> > > > > filled so the heads flew lower.

So the helium made the disks "drop off" the SATA cables?
How does that work?

> > > > https://www.howtogeek.com/803276/cmr-vs.-smr-hard-drives-whats-the-difference/
> > > 
> > > I carefully note, the use of Helium and its problems is very carefully
> > > ignored.

What's the connection between shingled disks and helium?

> > Western Digital at least claims to have solved the leaking
> > problem with helium and since they've been making those drives for over
> > a decade, I think it's solved.

When were these leakage incidents? I haven't heard about them;
only Gene's wartime anecdotes about helium passing through inch-thick
walls of Monel with impunity.

> Your source for this?

Indeed, for any of this. As for facts from the internet, I read
this on one page selected at random — well, a top google hit:

  
https://recoverysquad.com.au/what-are-the-advantages-of-helium-sealed-hard-drives/

 "Helium HDDs as compared to standard HDDs

  Helium HDDs offer several advantages over traditional hard drives,
  including:

  [ … ]

  · Their lower power consumption translates into longer battery life for 
portable devices.
  ↑↑↑
  · And lastly, they generate less heat, which can be an issue with traditional 
HDDs.
 ↑↑
  [ … ]

  What are the challenges with Helium Hard Drives?

  The challenges with helium hard drives are that they are expensive to produce 
and
  they have a limited lifespan. The drives tend to run a bit hotter than 
traditional
  hard drives, and they also use more power."  
 ↑↑

WTF?

Cheers,
David.

Re: smartctl cannot access my storage, need syntax help

[Anssi Saari]

"Roy J. Tellason, Sr."  writes:

> On Thursday 25 January 2024 09:03:36 am Anssi Saari wrote:
>> Western Digital at least claims to have solved the leaking
>> problem with helium and since they've been making those drives for over
>> a decade, I think it's solved.
>
> Your source for this?

The internet, of course. WDs blog, Backblaze, a bunch of news sites,
basically whatever Qwant coughed up. Why?

Re: smartctl cannot access my storage, need syntax help

[Roy J. Tellason, Sr.]

On Thursday 25 January 2024 09:03:36 am Anssi Saari wrote:
> Western Digital at least claims to have solved the leaking
> problem with helium and since they've been making those drives for over
> a decade, I think it's solved.

Your source for this?

-- 
Member of the toughest, meanest, deadliest, most unrelenting -- and
ablest -- form of life in this section of space,  a critter that can
be killed but can't be tamed.  --Robert A. Heinlein, "The Puppet Masters"
-
Information is more dangerous than cannon to a society ruled by lies. --James 
M Dakin

Re: smartctl cannot access my storage, need syntax help

[Anssi Saari]

gene heskett  writes:

> I carefully note, the use of Helium and its problems is very carefully
> ignored.

I suppose helium is not required for SMR drives and could be used in CMR
drives too... Western Digital at least claims to have solved the leaking
problem with helium and since they've been making those drives for over
a decade, I think it's solved.

Really a shame how the whole SMR mess happened. For the last decade or
so I thought I can just double the size of my little "stuff" server
(mirrored 1 TB drives) by bigger 2.5" drives. But no, first they came
out with fat 2 TB drives and then the slim drives all went SMR. There
was a time when suitable drives were available but I wasn't shopping
then. So instead of replacing just the drives, I replaced
everything. The new server is way faster and has a lot more storage but
it's also noisier and slower to wake due to the 3.5" drives.

Re: Powered USB hub [was: Re: smartctl cannot access my storage, need syntax help]

[Anssi Saari]

Max Nikulin  writes:

> Purchasing a powered USB hub, I made a mistake. I have not checked
> compatibility with hubctl in advance.
> https://github.com/mvp/uhubctl/

Wow, that's very cool. I wonder if there's anything similar for USB
switches? I have one that's software controllable but it's not great for
my purpose, which is doing the K and M parts of a poor man's KVM switch.

Re: smartctl cannot access my storage, need syntax help

[Karl Vogel]

On Tue, Jan 23, 2024 at 06:05:29AM -0500, gene heskett wrote:
> On 1/23/24 00:30, Karl Vogel wrote:
> >>> On 1/22/24 11:31, gene heskett wrote:
> >
> > G> How does an 8T backup server sound for another $200 in hdwe?  Very
> > G> enticing and I do have the sheckel's.
> >
> > https://www.amazon.com/dp/B07CQJBSQL
> > Seagate Desktop 8TB external Hard Drive, 3.5 Inch, USB 3.0 STGY8000400
> > $168.18
> >
> > What if you buy two, use one for a complete backup and the other for
> > incrementals or differentials?  (I know, more than $200...)
> 
> My disastrous experience with the last pair of seagates preclude exploring
> that path, ever again.

Sorry, the Seagate was just an example -- I prefer Western Digital myself.
My only point was using one or two external drives for backups.

-- 
Karl Vogel  I don't speak for anyone but myself

And as we all know from experiments conducted during the Korean War, Diane,
sleep deprivation is a one-way ticket to temporary psychosis.
--FBI Special Agent Dale Cooper, "Twin Peaks"

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/22/24 22:48, Stefan Monnier wrote:

some sort of 2T SSD's that comes as a usb-c drive, skipping the sata
convertor entirely at $27/copy. If it works as an 8T lvm with a 2T holding


AFAIK 2T for $27 doesn't exist yet in the current real world.
You can find a fair number of creatively sized USB disks in that price
range, but they are lying (tho admittedly, they tend to lye in a more
obvious way with claimed capacities of 30TB or even more).

This doesn't pass my scam detector.  I hope it's wrong for once.

So do I Stefan. But somebody had to test and report.  And I won't starve 
if t fails.


 Stefan

.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/23/24 06:12, Gremlin wrote:

On 1/23/24 06:04, gene heskett wrote:

On 1/23/24 00:30, Karl Vogel wrote:

On 1/22/24 11:31, gene heskett wrote:


G> How does an 8T backup server sound for another $200 in hdwe?  Very
G> enticing and I do have the sheckel's.

https://www.amazon.com/dp/B07CQJBSQL
Seagate Desktop 8TB external Hard Drive, 3.5 Inch, USB 3.0 STGY8000400
$168.18

What if you buy two, use one for a complete backup and the other for
incrementals or differentials?  (I know, more than $200...)

My disastrous experience with the last pair of seagates preclude 
exploring that path, ever again. I bought a couple 2T's to replace  2 
1T's I had outgrown and had close to 70,000 spining hours on the.  
They lasted a bit less than 30 days, dropping off the sata cables 
connecting then, never to be found again. Then I find they were 
shingled tech, and helium filled so the heads flew lower.



https://www.howtogeek.com/803276/cmr-vs.-smr-hard-drives-whats-the-difference/


I carefully note, the use of Helium and its problems is very carefully 
ignored.



.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[Gremlin]

On 1/23/24 06:04, gene heskett wrote:

On 1/23/24 00:30, Karl Vogel wrote:

On 1/22/24 11:31, gene heskett wrote:


G> How does an 8T backup server sound for another $200 in hdwe?  Very
G> enticing and I do have the sheckel's.

https://www.amazon.com/dp/B07CQJBSQL
Seagate Desktop 8TB external Hard Drive, 3.5 Inch, USB 3.0 STGY8000400
$168.18

What if you buy two, use one for a complete backup and the other for
incrementals or differentials?  (I know, more than $200...)

My disastrous experience with the last pair of seagates preclude 
exploring that path, ever again. I bought a couple 2T's to replace  2 
1T's I had outgrown and had close to 70,000 spining hours on the.  They 
lasted a bit less than 30 days, dropping off the sata cables connecting 
then, never to be found again. Then I find they were shingled tech, and 
helium filled so the heads flew lower.



https://www.howtogeek.com/803276/cmr-vs.-smr-hard-drives-whats-the-difference/

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/23/24 02:31, David Christensen wrote:

On 1/22/24 19:55, gene heskett wrote:

On 1/22/24 21:59, David Christensen wrote:

On 1/22/24 18:44, gene heskett wrote:

On 1/22/24 18:46, David Christensen wrote:

On 1/22/24 11:31, gene heskett wrote:
How does an 8T backup server sound for another $200 in hdwe?  Very 
enticing and I do have the sheckel's.


What hardware?

I clicked on place order, for a 7 port powered usb3 hub, and a 6 
pack of some sort of 2T SSD's that comes as a usb-c drive, skipping 
the sata convertor entirely at $27/copy. If it works as an 8T lvm 
with a 2T holding disk from 5 of them, fine, else it is experience, 
which according to TANSTAAFL, costs money.  Typical of amazon, 
though the drives will be drop shipped from China, arriving Feb. 1 
at earliest.


UNK is if the drives come with an A to C cable, 6" long would be 
sufficient.


URL's?


hub:





Okay.



2T ssd's:





Please use a tool to confirm that the drives actually store 2 TB when 
you receive them.



At that price, you betcha.


David

.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/23/24 00:30, Karl Vogel wrote:

On 1/22/24 11:31, gene heskett wrote:


G> How does an 8T backup server sound for another $200 in hdwe?  Very
G> enticing and I do have the sheckel's.

https://www.amazon.com/dp/B07CQJBSQL
Seagate Desktop 8TB external Hard Drive, 3.5 Inch, USB 3.0 STGY8000400
$168.18

What if you buy two, use one for a complete backup and the other for
incrementals or differentials?  (I know, more than $200...)

My disastrous experience with the last pair of seagates preclude 
exploring that path, ever again. I bought a couple 2T's to replace  2 
1T's I had outgrown and had close to 70,000 spining hours on the.  They 
lasted a bit less than 30 days, dropping off the sata cables connecting 
then, never to be found again. Then I find they were shingled tech, and 
helium filled so the heads flew lower.


But I have experience with helium in large car load qty's of it and in 
30 days I can guarantee those drives no longer had any Helium in them if 
there was any positive pressure inside them. This was about a year after 
JFK launched his effort to put men on the moon the first time.


A bank of 16 monel bottles, each about a foot in diameter and maybe 12 
feet long, pumped up to about 7500 psia for the night at midnight at 
Stellardyne labs in Sandy Eggo while testing the atlas stuff that gave 
John Glenn his 1st full orbital ride, were down to around 5800 psia at 8 
the next morning when the day shift got there. The Helium atom is so 
small it creeps between the atoms of tha monel, the best alloy for that. 
and heads for outer space never to be seen again.


And that was considered normal, and better thn the 50,000 gallon steel 
tank that caught and recyled the test cells output, we had a 6 stage 
cardox compress in the back yard that suck from that big tank and put it 
back on those monel bottles. With a guvmnt truck bringing in new to 
replace it 2x a week. In those days the guv automatically owned any 
helium recovered from deep in mines by air reduction techniques.  It's a 
precious commodity, worth close to $100,000 a truckload in $1.75 min 
wage days. When its gone, its gone and we will exhaust the planets 
supply by filling party balloons with it. Filling hard drives with it to 
make the heads fly lower is idiocy. Regardless of how thick you cast the 
alu or pot metal of a drive housing, it will be gone much quicker than 
those monel bottles allowed.


They (Seagate) could have gotten a longer term lower head fly by sealing 
them up in a vacuum equ to the top of Everest.


Like Albert Eintsein said, the 2 most common things in the universe are 
stupidity and hydrogen, in that order.


You may not have arrived yet Karl, but I was there, busy testing the 
fuel pressure regulators for Atlas rockets and keeping my wife barefoot 
and pregnant, her choice for both.


Take care, stay warm, dry and well Karl.

Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/22/24 19:55, gene heskett wrote:

On 1/22/24 21:59, David Christensen wrote:

On 1/22/24 18:44, gene heskett wrote:

On 1/22/24 18:46, David Christensen wrote:

On 1/22/24 11:31, gene heskett wrote:
How does an 8T backup server sound for another $200 in hdwe?  Very 
enticing and I do have the sheckel's.


What hardware?

I clicked on place order, for a 7 port powered usb3 hub, and a 6 pack 
of some sort of 2T SSD's that comes as a usb-c drive, skipping the 
sata convertor entirely at $27/copy. If it works as an 8T lvm with a 
2T holding disk from 5 of them, fine, else it is experience, which 
according to TANSTAAFL, costs money.  Typical of amazon, though the 
drives will be drop shipped from China, arriving Feb. 1 at earliest.


UNK is if the drives come with an A to C cable, 6" long would be 
sufficient.


URL's?


hub:





Okay.



2T ssd's:





Please use a tool to confirm that the drives actually store 2 TB when 
you receive them.



David

Re: smartctl cannot access my storage, need syntax help

[Karl Vogel]

>> On 1/22/24 11:31, gene heskett wrote:

G> How does an 8T backup server sound for another $200 in hdwe?  Very
G> enticing and I do have the sheckel's.

https://www.amazon.com/dp/B07CQJBSQL
Seagate Desktop 8TB external Hard Drive, 3.5 Inch, USB 3.0 STGY8000400
$168.18

What if you buy two, use one for a complete backup and the other for
incrementals or differentials?  (I know, more than $200...)

-- 
Karl Vogel  I don't speak for anyone but myself

I yam Popeye of Borg.  You will be askimilgrated.

Re: Powered USB hub [was: Re: smartctl cannot access my storage, needsyntax help]

[gene heskett]

On 1/22/24 23:10, Max Nikulin wrote:

On 23/01/2024 10:55, gene heskett wrote:

hub:




Purchasing a powered USB hub, I made a mistake. I have not checked 
compatibility with hubctl in advance.

https://github.com/mvp/uhubctl/

.

A valuable tool indeed, bookmarked FFR.  Thank you Max.
Take care, stay warm, dry and well.

Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Powered USB hub [was: Re: smartctl cannot access my storage, need syntax help]

[Max Nikulin]

On 23/01/2024 10:55, gene heskett wrote:

hub:




Purchasing a powered USB hub, I made a mistake. I have not checked 
compatibility with hubctl in advance.

https://github.com/mvp/uhubctl/

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/22/24 21:59, David Christensen wrote:

On 1/22/24 18:44, gene heskett wrote:

On 1/22/24 18:46, David Christensen wrote:

On 1/22/24 11:31, gene heskett wrote:
How does an 8T backup server sound for another $200 in hdwe?  Very 
enticing and I do have the sheckel's.


What hardware?

I clicked on place order, for a 7 port powered usb3 hub, and a 6 pack 
of some sort of 2T SSD's that comes as a usb-c drive, skipping the 
sata convertor entirely at $27/copy. If it works as an 8T lvm with a 
2T holding disk from 5 of them, fine, else it is experience, which 
according to TANSTAAFL, costs money.  Typical of amazon, though the 
drives will be drop shipped from China, arriving Feb. 1 at earliest.


UNK is if the drives come with an A to C cable, 6" long would be 
sufficient.



URL's?


hub:



2T ssd's:




David


.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[Stefan Monnier]

> some sort of 2T SSD's that comes as a usb-c drive, skipping the sata
> convertor entirely at $27/copy. If it works as an 8T lvm with a 2T holding

AFAIK 2T for $27 doesn't exist yet in the current real world.
You can find a fair number of creatively sized USB disks in that price
range, but they are lying (tho admittedly, they tend to lye in a more
obvious way with claimed capacities of 30TB or even more).

This doesn't pass my scam detector.  I hope it's wrong for once.


Stefan

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/22/24 18:44, gene heskett wrote:

On 1/22/24 18:46, David Christensen wrote:

On 1/22/24 11:31, gene heskett wrote:
How does an 8T backup server sound for another $200 in hdwe?  Very 
enticing and I do have the sheckel's.


What hardware?

I clicked on place order, for a 7 port powered usb3 hub, and a 6 pack of 
some sort of 2T SSD's that comes as a usb-c drive, skipping the sata 
convertor entirely at $27/copy. If it works as an 8T lvm with a 2T 
holding disk from 5 of them, fine, else it is experience, which 
according to TANSTAAFL, costs money.  Typical of amazon, though the 
drives will be drop shipped from China, arriving Feb. 1 at earliest.


UNK is if the drives come with an A to C cable, 6" long would be 
sufficient.



URL's?


David

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/22/24 18:46, David Christensen wrote:

On 1/22/24 11:31, gene heskett wrote:

On 1/22/24 12:54, David Christensen wrote:
Perhaps it is time to switch to another backup system, or build your 
own.

..
That I'm contemplating, using a pi clone but still running the amanda 
I just installed all 3 debs of on a bananapi-m5. 



Okay.


How does an 8T backup server sound for another $200 in hdwe?  Very 
enticing and I do have the sheckel's.



What hardware?

I clicked on place order, for a 7 port powered usb3 hub, and a 6 pack of 
some sort of 2T SSD's that comes as a usb-c drive, skipping the sata 
convertor entirely at $27/copy. If it works as an 8T lvm with a 2T 
holding disk from 5 of them, fine, else it is experience, which 
according to TANSTAAFL, costs money.  Typical of amazon, though the 
drives will be drop shipped from China, arriving Feb. 1 at earliest.


UNK is if the drives come with an A to C cable, 6" long would be sufficient.

Take care, stay warm, dry & well, David.

Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/22/24 11:31, gene heskett wrote:

On 1/22/24 12:54, David Christensen wrote:

Perhaps it is time to switch to another backup system, or build your own.
..
That I'm contemplating, using a pi clone but still running the amanda I 
just installed all 3 debs of on a bananapi-m5. 



Okay.


How does an 8T backup 
server sound for another $200 in hdwe?  Very enticing and I do have the 
sheckel's.



What hardware?


David

Re: smartctl cannot access my storage, need syntax help

[Stefan Monnier]

> That I'm contemplating, using a pi clone but still running the amanda I just
> installed all 3 debs of on a bananapi-m5.  How does an 8T backup server
> sound for another $200 in hdwe?  Very enticing and I do have the sheckel's.

I remember Amanda fondly from the days when I was backing up a labs
machine to DATs.  And while I see the benefit of using a tool with which
you're familiar, you might want to look at alternatives like Bup, Borg,
Restic, Bupstash, ... (or even Rsync/Rsnapshot).

Amanda was designed specifically to work efficiently when backing up to
a tape device.  If, like most people nowadays, you backup to something
like an HDD or SSD, Amanda is IMO overly complex and slow and doesn't
give you as much functionality as more modern alternatives (alternatives
which simply wouldn't work when backing up to a tape).


Stefan

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/22/24 12:54, David Christensen wrote:

On 1/22/24 03:23, gene heskett wrote:

On 1/22/24 04:46, David Christensen wrote:

It appears Amanda has a script API for both the client and the server:

https://manpages.debian.org/buster/amanda-common/amanda-scripts.7.en.html
...
All this is possible David, but needs someone to do it. So far our 
list of volunteers is pretty slim. I once wrote a script that added 
amanda's database to the end of the vtape amanda had just made, making 
a bare metal recovery to the state it had just reported instead of a 
bare metal being one run out of date, but I did that in bash. I've 
never did anything to amanda itself except compile it, its old perl, 
old python and probably older bash, all dumped into the same bowl and 
the mixer turned on high. Amanda, right now, needs 15 years of catchup 
tlc. I haven't even tried to build it since wheezy and I have far 
newer srcs than the current 3.51 here. While I have the /home/amanda 
dir with all that it , i'd have to create anew amanda user with 
passwd-less access to the system to even attempt a build of what I have.



Perhaps it is time to switch to another backup system, or build your own.


David

That I'm contemplating, using a pi clone but still running the amanda I 
just installed all 3 debs of on a bananapi-m5.  How does an 8T backup 
server sound for another $200 in hdwe?  Very enticing and I do have the 
sheckel's.

.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/22/24 03:23, gene heskett wrote:

On 1/22/24 04:46, David Christensen wrote:

It appears Amanda has a script API for both the client and the server:

https://manpages.debian.org/buster/amanda-common/amanda-scripts.7.en.html
...
All this is possible David, but needs someone to do it. So far our list 
of volunteers is pretty slim. I once wrote a script that added amanda's 
database to the end of the vtape amanda had just made, making a bare 
metal recovery to the state it had just reported instead of a bare metal 
being one run out of date, but I did that in bash. I've never did 
anything to amanda itself except compile it, its old perl, old python 
and probably older bash, all dumped into the same bowl and the mixer 
turned on high. Amanda, right now, needs 15 years of catchup tlc. I 
haven't even tried to build it since wheezy and I have far newer srcs 
than the current 3.51 here. While I have the /home/amanda dir with all 
that it , i'd have to create anew amanda user with passwd-less access to 
the system to even attempt a build of what I have.



Perhaps it is time to switch to another backup system, or build your own.


David

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/22/24 04:46, David Christensen wrote:

On 1/21/24 21:42, gene heskett wrote:

On 1/21/24 18:29, David Christensen wrote:

On 1/21/24 14:48, gene heskett wrote:

On 1/21/24 16:13, David Christensen wrote:

On 1/21/24 03:47, gene heskett wrote:

On 1/21/24 01:33, David Christensen wrote:
3.  For Amanda, either add more HDD's to the storage server or 
build another server.  If another server, shut it down when you are 
not using it.

...
Designed to run every night when things are relatively quiet, how 
this works well depends on the other machines it is backing up be 
available.

...
1. Wake on LAN.

2. Wake at preset day/time.

...
Unfortunately, amanda is truely ancient, for some reason the 
originator who first wrote it in the later 70's IIRC sold it nearly 20 
years ago to a commercial outfit called zmanda, who took it more or 
less commercial, throwing the amanda named version under the buss. 
They must have ran out of money and resold it to another outfit, who 
has redoubled their effort to get rid of the free version. One of the 
things it has not been fixed to do, is issue a wakeup call, and wait 
for the clients to get their stuff in one sock, say 30 seconds to get 
everything spun up and ready to take orders, so I'm pretty sure a 
client that doesn't respond in milliseconds will be skipped. So 
basically, amanda needs to be officially forked since the current 
owner, Betsol has not made any contribution to amanda that amounts to 
an actual update but once in 6 or 7 years now,  There's several things 
it now needs, such as the wake-on-lan support done right. Python is 
part of it but python 2 is still needed. Or a whole new start for 
something to replace and put it back squarely in the gplv2 or 3 camp. 
If there is actually another capable of diddling the level schedule 
like amanda does, I'm sure I could name some of the major users that 
would jump ship in a week or so once they became aware of a workalike.



It appears Amanda has a script API for both the client and the server:

https://manpages.debian.org/buster/amanda-common/amanda-scripts.7.en.html


The zmanda wiki has a Script API page, but it is empty (?):

https://wiki.zmanda.com/index.php/Script_API.


For BIOS/UEFI wake-on-lan, it might be possible to write a script that 
wakes the clients, to write a script that shuts down the clients, to 
configure the Amanda server to run the wake script before backups, and 
to configure the Amanda server to run the shutdown script after backups.



For BIOS/UEFI wake at preset day/time, it might be possible to set the 
clients to wake before the scheduled backup time, to write a script that 
shuts down the clients, and to configure the Amanda server to run the 
shutdown script after backups.



David
All this is possible David, but needs someone to do it. So far our list 
of volunteers is pretty slim. I once wrote a script that added amanda's 
database to the end of the vtape amanda had just made, making a bare 
metal recovery to the state it had just reported instead of a bare metal 
being one run out of date, but I did that in bash. I've never did 
anything to amanda itself except compile it, its old perl, old python 
and probably older bash, all dumped into the same bowl and the mixer 
turned on high. Amanda, right now, needs 15 years of catchup tlc. I 
haven't even tried to build it since wheezy and I have far newer srcs 
than the current 3.51 here. While I have the /home/amanda dir with all 
that it , i'd have to create anew amanda user with passwd-less access to 
the system to even attempt a build of what I have.


.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/21/24 21:42, gene heskett wrote:

On 1/21/24 18:29, David Christensen wrote:

On 1/21/24 14:48, gene heskett wrote:

On 1/21/24 16:13, David Christensen wrote:

On 1/21/24 03:47, gene heskett wrote:

On 1/21/24 01:33, David Christensen wrote:
3.  For Amanda, either add more HDD's to the storage server or build 
another server.  If another server, shut it down when you are not 
using it.

...
Designed to run every night when things are relatively quiet, 
how this works well depends on the other machines it is backing up be 
available.

...
1. Wake on LAN.

2. Wake at preset day/time.

...
Unfortunately, amanda is truely ancient, for some reason the originator 
who first wrote it in the later 70's IIRC sold it nearly 20 years ago to 
a commercial outfit called zmanda, who took it more or less commercial, 
throwing the amanda named version under the buss. They must have ran out 
of money and resold it to another outfit, who has redoubled their effort 
to get rid of the free version. One of the things it has not been fixed 
to do, is issue a wakeup call, and wait for the clients to get their 
stuff in one sock, say 30 seconds to get everything spun up and ready to 
take orders, so I'm pretty sure a client that doesn't respond in 
milliseconds will be skipped. So basically, amanda needs to be 
officially forked since the current owner, Betsol has not made any 
contribution to amanda that amounts to an actual update but once in 6 or 
7 years now,  There's several things it now needs, such as the 
wake-on-lan support done right. Python is part of it but python 2 is 
still needed. Or a whole new start for something to replace and put it 
back squarely in the gplv2 or 3 camp. If there is actually another 
capable of diddling the level schedule like amanda does, I'm sure I 
could name some of the major users that would jump ship in a week or so 
once they became aware of a workalike.



It appears Amanda has a script API for both the client and the server:

https://manpages.debian.org/buster/amanda-common/amanda-scripts.7.en.html


The zmanda wiki has a Script API page, but it is empty (?):

https://wiki.zmanda.com/index.php/Script_API.


For BIOS/UEFI wake-on-lan, it might be possible to write a script that 
wakes the clients, to write a script that shuts down the clients, to 
configure the Amanda server to run the wake script before backups, and 
to configure the Amanda server to run the shutdown script after backups.



For BIOS/UEFI wake at preset day/time, it might be possible to set the 
clients to wake before the scheduled backup time, to write a script that 
shuts down the clients, and to configure the Amanda server to run the 
shutdown script after backups.



David

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/21/24 18:29, David Christensen wrote:

On 1/21/24 14:48, gene heskett wrote:

On 1/21/24 16:13, David Christensen wrote:

On 1/21/24 03:47, gene heskett wrote:

On 1/21/24 01:33, David Christensen wrote:
I am still uncertain if those are internal SSD errors or SATA 
errors. Please check if you see matching errors in dmesg(1).


There aren't any. Those hours would very closely correspond to my 
attempts to rsync and the OOM deamon killed the machine, which it 
did around 10 times.  So logging by then had been killed. That to me 
is the smoking gun. 



Kernel ring buffer is renewed with each boot and newer messages 
overwrite older messages.  So, you will want to save or clear the 
ring buffer with demsg(1), save a SMART full report, exercise the 
disk with dd(1) and/or a SMART test, save the ring buffer, save a 
SMART full report, and analyze everything to see if you have disk 
problems, SATA problems, and/or system problems.  Once everything 
passes without error, the disk is ready to be put into service.



2T is enough /home for the nonce. so I'll do the rsync thing going 
the other direction, using it for a backup of /home until I'm ready 
for trixie.


However I am tempted to zero the drives an recreate the raid w/o 
formatting since the mdadm seems capable to installing itw own 
filesystems to use the whole drive unpartitioned, giving me a backup 
that sizewise is about the same as the single 2T drive has now.


And although my single experience with lvm over a decade ago was a 
total disaster, made out of used spinning rust I may now see how the 
other 4 2T's assembled as a lvm for amandas vtapes as an 8T lvm to 
backup the whole system, which in addition to the 4 cnc'd machines, 
has over the last 5 years seen a train of 3d printers go by. If all 
3, currently a WIP, get rebuilt, the smallest is 305 by, the largest 
is 400 by.  And all I hope will lay plastic at 200+ mm a second. 
Normal consumer stuff is 40 to 60.


Obviously I have an eclectic choice of too many hobbies. ;o)>
Now if curiosity doesn't kill this cat, I need to find some 
breakfast and git to it.



This and other threads have led me to the conclusion that consumer 
SSD's are meant for devices that are off most of the time -- e.g. 
notepad, laptop, desktop, and workstation computers.  If you put them 
into a NAS/ file server and run them 24x7, they will die sometime 
after 2 years.


That has not been my experience at all David, I bought a 4 pack of 
120G ssd's when they were the biggest available and replaced 3 
spinning rust drives that had 50-70k hours on them with these. My cnc 
machines are all wired so power for the mill/lathe/what have you is 
totally controlled by the enable key, f2, so if f2 is off only the 
computer is running. That was at least 6 years ago. Then I installed a 
240G as an extra drive on the rpi4 that runs my biggest lathe and made 
a buildbot out of it to pull linuxcnc-master from github and build it, 
also armhf kernels for linuxcnc's realtime needs.  The 120G 
disappeared in about a year, replaced the adapter with a startech, 
drive was and is just fine. There is now at least 5 years on everyone 
of those original 120's, zero SSD problems in the whole lot.



I also have small SSD's that have lasted far longer than 2 years on 
mixed duty, including 24x7 (Intel SSD 520 Series 60 GB).  The relevant 
recent threads on this list seem to be 1+ TB Samsung's.



It is interesting to note that BackBlaze does not seem to use Samsung 
SSD's:


https://www.backblaze.com/blog/ssd-edition-2023-mid-year-drive-stats-review/


3.  For Amanda, either add more HDD's to the storage server or build 
another server.  If another server, shut it down when you are not 
using it.



Speaking as someone who has used amanda for about 25 years:

People don't always understand that one of Amanda prime directives is 
to balance the size of an individual back up run by advancing the 
level 3 scheduled for tonight, by advancing it to level 0 if this run 
is only going to be small. The only guarantee is that if you have a 10 
day schedule, all machines/dle's, will get that level 0 backup not 
more than 10 days after the last one.   You choose how many days long 
that cycle is. I adjust it so the storage is around 75 to 80% used 
after the schedule has stabilized. This may take quite  a few such 
cycles. Designed to run every night when things are relatively quiet, 
how this works well depends on the other machines it is backing up be 
available.


Machines missing at backup time can and will muck things up for this 
efficient scheduling. Corporate users of Amanda, used to doing it 
their way, backing the weeks business on friday nights just don't 
understand that the Amanda way gets them a 100% coverage backup by 
backing up only the differences from the previous run of that dle 
every night is far superior to their fridey night when most of the 
offices machines are turned off for the weekend. For those cases we 
recommend 

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/21/24 14:48, gene heskett wrote:

On 1/21/24 16:13, David Christensen wrote:

On 1/21/24 03:47, gene heskett wrote:

On 1/21/24 01:33, David Christensen wrote:
I am still uncertain if those are internal SSD errors or SATA 
errors. Please check if you see matching errors in dmesg(1).


There aren't any. Those hours would very closely correspond to my 
attempts to rsync and the OOM deamon killed the machine, which it did 
around 10 times.  So logging by then had been killed. That to me is 
the smoking gun. 



Kernel ring buffer is renewed with each boot and newer messages 
overwrite older messages.  So, you will want to save or clear the ring 
buffer with demsg(1), save a SMART full report, exercise the disk with 
dd(1) and/or a SMART test, save the ring buffer, save a SMART full 
report, and analyze everything to see if you have disk problems, SATA 
problems, and/or system problems.  Once everything passes without 
error, the disk is ready to be put into service.



2T is enough /home for the nonce. so I'll do the rsync thing going 
the other direction, using it for a backup of /home until I'm ready 
for trixie.


However I am tempted to zero the drives an recreate the raid w/o 
formatting since the mdadm seems capable to installing itw own 
filesystems to use the whole drive unpartitioned, giving me a backup 
that sizewise is about the same as the single 2T drive has now.


And although my single experience with lvm over a decade ago was a 
total disaster, made out of used spinning rust I may now see how the 
other 4 2T's assembled as a lvm for amandas vtapes as an 8T lvm to 
backup the whole system, which in addition to the 4 cnc'd machines, 
has over the last 5 years seen a train of 3d printers go by. If all 
3, currently a WIP, get rebuilt, the smallest is 305 by, the largest 
is 400 by.  And all I hope will lay plastic at 200+ mm a second.  
Normal consumer stuff is 40 to 60.


Obviously I have an eclectic choice of too many hobbies. ;o)>
Now if curiosity doesn't kill this cat, I need to find some breakfast 
and git to it.



This and other threads have led me to the conclusion that consumer 
SSD's are meant for devices that are off most of the time -- e.g. 
notepad, laptop, desktop, and workstation computers.  If you put them 
into a NAS/ file server and run them 24x7, they will die sometime 
after 2 years.


That has not been my experience at all David, I bought a 4 pack of 120G 
ssd's when they were the biggest available and replaced 3 spinning rust 
drives that had 50-70k hours on them with these. My cnc machines are all 
wired so power for the mill/lathe/what have you is totally controlled by 
the enable key, f2, so if f2 is off only the computer is running. That 
was at least 6 years ago. Then I installed a 240G as an extra drive on 
the rpi4 that runs my biggest lathe and made a buildbot out of it to 
pull linuxcnc-master from github and build it, also armhf kernels for 
linuxcnc's realtime needs.  The 120G disappeared in about a year, 
replaced the adapter with a startech, drive was and is just fine. There 
is now at least 5 years on everyone of those original 120's, zero SSD 
problems in the whole lot.



I also have small SSD's that have lasted far longer than 2 years on 
mixed duty, including 24x7 (Intel SSD 520 Series 60 GB).  The relevant 
recent threads on this list seem to be 1+ TB Samsung's.



It is interesting to note that BackBlaze does not seem to use Samsung SSD's:

https://www.backblaze.com/blog/ssd-edition-2023-mid-year-drive-stats-review/


3.  For Amanda, either add more HDD's to the storage server or build 
another server.  If another server, shut it down when you are not 
using it.



Speaking as someone who has used amanda for about 25 years:

People don't always understand that one of Amanda prime directives is to 
balance the size of an individual back up run by advancing the level 3 
scheduled for tonight, by advancing it to level 0 if this run is only 
going to be small. The only guarantee is that if you have a 10 day 
schedule, all machines/dle's, will get that level 0 backup not more than 
10 days after the last one.   You choose how many days long that cycle 
is. I adjust it so the storage is around 75 to 80% used after the 
schedule has stabilized. This may take quite  a few such cycles. 
Designed to run every night when things are relatively quiet, how this 
works well depends on the other machines it is backing up be available.


Machines missing at backup time can and will muck things up for this 
efficient scheduling. Corporate users of Amanda, used to doing it their 
way, backing the weeks business on friday nights just don't understand 
that the Amanda way gets them a 100% coverage backup by backing up only 
the differences from the previous run of that dle every night is far 
superior to their fridey night when most of the offices machines are 
turned off for the weekend. For those cases we recommend composing two 
or more dle files and rigging cron to 

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/21/24 16:13, David Christensen wrote:

On 1/21/24 03:47, gene heskett wrote:

On 1/21/24 01:33, David Christensen wrote:
I am still uncertain if those are internal SSD errors or SATA errors. 
Please check if you see matching errors in dmesg(1).


There aren't any. Those hours would very closely correspond to my 
attempts to rsync and the OOM deamon killed the machine, which it did 
around 10 times.  So logging by then had been killed. That to me is 
the smoking gun. 



Kernel ring buffer is renewed with each boot and newer messages 
overwrite older messages.  So, you will want to save or clear the ring 
buffer with demsg(1), save a SMART full report, exercise the disk with 
dd(1) and/or a SMART test, save the ring buffer, save a SMART full 
report, and analyze everything to see if you have disk problems, SATA 
problems, and/or system problems.  Once everything passes without error, 
the disk is ready to be put into service.



2T is enough /home for the nonce. so I'll do the rsync thing going the 
other direction, using it for a backup of /home until I'm ready for 
trixie.


However I am tempted to zero the drives an recreate the raid w/o 
formatting since the mdadm seems capable to installing itw own 
filesystems to use the whole drive unpartitioned, giving me a backup 
that sizewise is about the same as the single 2T drive has now.


And although my single experience with lvm over a decade ago was a 
total disaster, made out of used spinning rust I may now see how the 
other 4 2T's assembled as a lvm for amandas vtapes as an 8T lvm to 
backup the whole system, which in addition to the 4 cnc'd machines, 
has over the last 5 years seen a train of 3d printers go by. If all 3, 
currently a WIP, get rebuilt, the smallest is 305 by, the largest is 
400 by.  And all I hope will lay plastic at 200+ mm a second.  Normal 
consumer stuff is 40 to 60.


Obviously I have an eclectic choice of too many hobbies. ;o)>
Now if curiosity doesn't kill this cat, I need to find some breakfast 
and git to it.



This and other threads have led me to the conclusion that consumer SSD's 
are meant for devices that are off most of the time -- e.g. notepad, 
laptop, desktop, and workstation computers.  If you put them into a NAS/ 
file server and run them 24x7, they will die sometime after 2 years.


That has not been my experience at all David, I bought a 4 pack of 120G 
ssd's when they were the biggest available and replaced 3 spinning rust 
drives that had 50-70k hours on them with these. My cnc machines are all 
wired so power for the mill/lathe/what have you is totally controlled by 
the enable key, f2, so if f2 is off only the computer is running. That 
was at least 6 years ago. Then I installed a 240G as an extra drive on 
the rpi4 that runs my biggest lathe and made a buildbot out of it to 
pull linuxcnc-master from github and build it, also armhf kernels for 
linuxcnc's realtime needs.  The 120G disappeared in about a year, 
replaced the adapter with a startech, drive was and is just fine. There 
is now at least 5 years on everyone of those original 120's, zero SSD 
problems in the whole lot.


So, I suggest:

1.  Build a storage server using NAS or enterprise HDD's.  Use an 
enterprise SSD or DOM for the OS.  Run it 24x7 or shut it down as you like.


2.  Use your Asus PRIME Z370-A II as a workstation.  Install the WD 
Black M.2 NVMe PCIe SSD.  Connect the optical drive to the first 
motherboard SATA port.  Install Debian onto the WD Black.  Then, connect 
the five Samsung EVO 870's to the remaining motherboard SATA ports.  Set 
them up as a 5-way mirror (RAID1).  Use the Samsung RAID as a scratch 
disk for your 3-D work.  As the Samsung's die off, replace them with the 
Gigastones.  Shut it down when you are not using it.


3.  For Amanda, either add more HDD's to the storage server or build 
another server.  If another server, shut it down when you are not using it.



Speaking as someone who has used amanda for about 25 years:

People don't always understand that one of Amanda prime directives is to 
balance the size of an individual back up run by advancing the level 3 
scheduled for tonight, by advancing it to level 0 if this run is only 
going to be small. The only guarantee is that if you have a 10 day 
schedule, all machines/dle's, will get that level 0 backup not more than 
10 days after the last one.   You choose how many days long that cycle 
is. I adjust it so the storage is around 75 to 80% used after the 
schedule has stabilized. This may take quite  a few such cycles. 
Designed to run every night when things are relatively quiet, how this 
works well depends on the other machines it is backing up be available.


Machines missing at backup time can and will muck things up for this 
efficient scheduling. Corporate users of Amanda, used to doing it their 
way, backing the weeks business on friday nights just don't understand 
that the Amanda way gets them a 100% coverage backup by backing up only 
the 

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/21/24 03:47, gene heskett wrote:

On 1/21/24 01:33, David Christensen wrote:
I am still uncertain if those are internal SSD errors or SATA errors. 
Please check if you see matching errors in dmesg(1).


There aren't any. Those hours would very closely correspond to my 
attempts to rsync and the OOM deamon killed the machine, which it did 
around 10 times.  So logging by then had been killed. That to me is the 
smoking gun. 



Kernel ring buffer is renewed with each boot and newer messages 
overwrite older messages.  So, you will want to save or clear the ring 
buffer with demsg(1), save a SMART full report, exercise the disk with 
dd(1) and/or a SMART test, save the ring buffer, save a SMART full 
report, and analyze everything to see if you have disk problems, SATA 
problems, and/or system problems.  Once everything passes without error, 
the disk is ready to be put into service.



2T is enough /home for the nonce. so I'll do the rsync 
thing going the other direction, using it for a backup of /home 
until I'm ready for trixie.


However I am tempted to zero the drives an recreate the raid w/o 
formatting since the mdadm seems capable to installing itw own 
filesystems to use the whole drive unpartitioned, giving me a backup 
that sizewise is about the same as the single 2T drive has now.


And although my single experience with lvm over a decade ago was a total 
disaster, made out of used spinning rust I may now see how the other 4 
2T's assembled as a lvm for amandas vtapes as an 8T lvm to backup the 
whole system, which in addition to the 4 cnc'd machines, has over the 
last 5 years seen a train of 3d printers go by. If all 3, currently a 
WIP, get rebuilt, the smallest is 305 by, the largest is 400 by.  And 
all I hope will lay plastic at 200+ mm a second.  Normal consumer stuff 
is 40 to 60.


Obviously I have an eclectic choice of too many hobbies. ;o)>
Now if curiosity doesn't kill this cat, I need to find some breakfast 
and git to it.



This and other threads have led me to the conclusion that consumer SSD's 
are meant for devices that are off most of the time -- e.g. notepad, 
laptop, desktop, and workstation computers.  If you put them into a NAS/ 
file server and run them 24x7, they will die sometime after 2 years.



So, I suggest:

1.  Build a storage server using NAS or enterprise HDD's.  Use an 
enterprise SSD or DOM for the OS.  Run it 24x7 or shut it down as you like.


2.  Use your Asus PRIME Z370-A II as a workstation.  Install the WD 
Black M.2 NVMe PCIe SSD.  Connect the optical drive to the first 
motherboard SATA port.  Install Debian onto the WD Black.  Then, connect 
the five Samsung EVO 870's to the remaining motherboard SATA ports.  Set 
them up as a 5-way mirror (RAID1).  Use the Samsung RAID as a scratch 
disk for your 3-D work.  As the Samsung's die off, replace them with the 
Gigastones.  Shut it down when you are not using it.


3.  For Amanda, either add more HDD's to the storage server or build 
another server.  If another server, shut it down when you are not using it.



David

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/21/24 04:35, Max Nikulin wrote:


On 21/01/2024 03:23, gene heskett wrote:

Right now nothing in the system is north of 32C, might get to 36C at
the end of a 9 minute build of something in OpenSCAD. 


I would say that 53°C and even 44°C is well above 36°C you expected:

On 21/01/2024 12:48, gene heskett wrote:


SCT Status Version:  3
SCT Version (vendor specific):   256 (0x0100)
Device State:    DST executing in background (3)
Current Temperature:    28 Celsius
Power Cycle Min/Max Temperature: 26/44 Celsius
Lifetime    Min/Max Temperature: 24/53 Celsius
Specified Max Operating Temperature:    70 Celsius
Under/Over Temperature Limit Count:   0/0



Device Statistics (GP Log 0x04)

0x05  =  =   =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1  28  ---  Current Temperature
0x05  0x020  1  53  ---  Highest Temperature
0x05  0x028  1  24  ---  Lowest Temperature
0x05  0x058  1  70  ---  Specified Maximum Operating 
Temperature


IIRC the fan in the front of an upper drive cage got unplugged for a 
while, half an hour maybe, about a year ago while I was doing my annual 
D on it.  These SSD's all of them have a label claiming they need 5 
volts and 1 amp, that is 5 watts, but I don't think that is a steady 
load, probably only when writing at 500+ mhz,  ! watt or less of heat is 
much closer to normal operation.


Thank you, take care, stay warm, dry and well, Max. Having a heat wave 
here, its up to 21F out at 12:25 pm here, 16" of white stuff on the 
front deck, got cold & had to replace the battery's in my smart t-stat 
about an hour ago.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/21/24 01:33, David Christensen wrote:

On 1/20/24 21:48, gene heskett wrote:

New -x version for this SSD attached


 > SMART Attributes Data Structure revision number: 1
 > Vendor Specific SMART Attributes with Thresholds:
 > ID# ATTRIBUTE_NAME  FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
 >   5 Reallocated_Sector_Ct   PO--CK   094   094   010    -    64
 > 183 Runtime_Bad_Block   PO--C-   094   094   010    -    64
 > 187 Uncorrectable_Error_Cnt -O--CK   099   099   000    -    392
 > 195 ECC_Error_Rate  -O-RC-   199   199   000    -    392
 > 199 CRC_Error_Count -OSRCK   099   099   000    -    2

Those attributes are worrisome.  Especially Reallocated_Sector_Ct and 
Runtime_Bad_Block -- I am confident those are inside the SSD.



 >   9 Power_On_Hours  -O--CK   095   095   000    -    21194

That is equivalent to 10.2 years at 40 hours/week.

Machine runs 24/7/365.25



 > 241 Total_LBAs_Written  -O--CK   099   099   000    -    38429262625

TBW specification for 1 TB drive is 600TB.  You are at 19.7.

relatively low IOW.



 > Error 466 [1] occurred at disk power-on lifetime: 21078 hours (878 
days + 6 hours)
 >   When the command that caused the error occurred, the device was 
active or idle.

 >
 >   After command completion occurred, registers were:
 >   ER -- ST COUNT  LBA_48  LH LM LL DV DC
 >   -- -- -- == -- == == == -- -- -- -- --
 >   40 -- 51 00 40 00 00 1b a4 0d 18 40 00  Error: WP at LBA = 
0x1ba40d18 = 463736088

 >
 >   Commands leading to the command that caused the error were:
 >   CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time 
Command/Feature_Name
 >   -- == -- == -- == == == -- -- -- -- --  --- 

 >   61 00 08 00 40 00 00 1b a4 0d 18 40 08  1d+03:35:20.430  WRITE 
FPDMA QUEUED
 >   60 0a 00 00 38 00 00 70 f1 a4 00 40 07  1d+03:35:20.430  READ FPDMA 
QUEUED
 >   60 07 80 00 30 00 00 70 f1 3c 80 40 06  1d+03:35:20.430  READ FPDMA 
QUEUED
 >   61 00 28 00 28 00 00 1b a4 0d 38 40 05  1d+03:35:20.430  WRITE 
FPDMA QUEUED
 >   47 00 00 00 01 00 00 00 00 00 00 40 02  1d+03:35:20.430  READ LOG 
DMA EXT

 >
 > Error 465 [0] occurred at disk power-on lifetime: 21078 hours (878 
days + 6 hours)

 > ...
 > Error 464 [3] occurred at disk power-on lifetime: 21078 hours (878 
days + 6 hours)

 > ...
 > Error 463 [2] occurred at disk power-on lifetime: 21078 hours (878 
days + 6 hours)


I am still uncertain if those are internal SSD errors or SATA errors. 
Please check if you see matching errors in dmesg(1).



There aren't any. Those hours would very closely correspond to my 
attempts to rsync and the OOM deamon killed the machine, which it did 
around 10 times.  So logging by then had been killed. That to me is the 
smoking gun. 2T is enough /home for the nonce. so I'll do the rsync 
thing going the other direction, using it for a backup of /home until 
I'm ready for trixie.


However I am tempted to zero the drives an recreate the raid w/o 
formatting since the mdadm seems capable to installing itw own 
filesystems to use the whole drive unpartitioned, giving me a backup 
that sizewise is about the same as the single 2T drive has now.


And although my single experience with lvm over a decade ago was a total 
disaster, made out of used spinning rust I may now see how the other 4 
2T's assembled as a lvm for amandas vtapes as an 8T lvm to backup the 
whole system, which in addition to the 4 cnc'd machines, has over the 
last 5 years seen a train of 3d printers go by. If all 3, currently a 
WIP, get rebuilt, the smallest is 305 by, the largest is 400 by.  And 
all I hope will lay plastic at 200+ mm a second.  Normal consumer stuff 
is 40 to 60.


Obviously I have an eclectic choice of too many hobbies. ;o)>
Now if curiosity doesn't kill this cat, I need to find some breakfast 
and git to it.


Thank you David, take care, stay warm dry and well.


David

.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[Max Nikulin]

On 21/01/2024 03:23, gene heskett wrote:

Right now nothing in the system is north of 32C, might get to 36C at
the end of a 9 minute build of something in OpenSCAD. 


I would say that 53°C and even 44°C is well above 36°C you expected:

On 21/01/2024 12:48, gene heskett wrote:


SCT Status Version:  3
SCT Version (vendor specific):   256 (0x0100)
Device State:DST executing in background (3)
Current Temperature:28 Celsius
Power Cycle Min/Max Temperature: 26/44 Celsius
LifetimeMin/Max Temperature: 24/53 Celsius
Specified Max Operating Temperature:70 Celsius
Under/Over Temperature Limit Count:   0/0



Device Statistics (GP Log 0x04)

0x05  =  =   =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1  28  ---  Current Temperature
0x05  0x020  1  53  ---  Highest Temperature
0x05  0x028  1  24  ---  Lowest Temperature
0x05  0x058  1  70  ---  Specified Maximum Operating Temperature

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/20/24 21:48, gene heskett wrote:

New -x version for this SSD attached


> SMART Attributes Data Structure revision number: 1
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME  FLAGSVALUE WORST THRESH FAIL RAW_VALUE
>   5 Reallocated_Sector_Ct   PO--CK   094   094   010-64
> 183 Runtime_Bad_Block   PO--C-   094   094   010-64
> 187 Uncorrectable_Error_Cnt -O--CK   099   099   000-392
> 195 ECC_Error_Rate  -O-RC-   199   199   000-392
> 199 CRC_Error_Count -OSRCK   099   099   000-2

Those attributes are worrisome.  Especially Reallocated_Sector_Ct and 
Runtime_Bad_Block -- I am confident those are inside the SSD.



>   9 Power_On_Hours  -O--CK   095   095   000-21194

That is equivalent to 10.2 years at 40 hours/week.


> 241 Total_LBAs_Written  -O--CK   099   099   000-38429262625

TBW specification for 1 TB drive is 600TB.  You are at 19.7.


> Error 466 [1] occurred at disk power-on lifetime: 21078 hours (878 
days + 6 hours)
>   When the command that caused the error occurred, the device was 
active or idle.

>
>   After command completion occurred, registers were:
>   ER -- ST COUNT  LBA_48  LH LM LL DV DC
>   -- -- -- == -- == == == -- -- -- -- --
>   40 -- 51 00 40 00 00 1b a4 0d 18 40 00  Error: WP at LBA = 
0x1ba40d18 = 463736088

>
>   Commands leading to the command that caused the error were:
>   CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time 
Command/Feature_Name
>   -- == -- == -- == == == -- -- -- -- --  --- 

>   61 00 08 00 40 00 00 1b a4 0d 18 40 08  1d+03:35:20.430  WRITE 
FPDMA QUEUED
>   60 0a 00 00 38 00 00 70 f1 a4 00 40 07  1d+03:35:20.430  READ FPDMA 
QUEUED
>   60 07 80 00 30 00 00 70 f1 3c 80 40 06  1d+03:35:20.430  READ FPDMA 
QUEUED
>   61 00 28 00 28 00 00 1b a4 0d 38 40 05  1d+03:35:20.430  WRITE 
FPDMA QUEUED
>   47 00 00 00 01 00 00 00 00 00 00 40 02  1d+03:35:20.430  READ LOG 
DMA EXT

>
> Error 465 [0] occurred at disk power-on lifetime: 21078 hours (878 
days + 6 hours)

> ...
> Error 464 [3] occurred at disk power-on lifetime: 21078 hours (878 
days + 6 hours)

> ...
> Error 463 [2] occurred at disk power-on lifetime: 21078 hours (878 
days + 6 hours)


I am still uncertain if those are internal SSD errors or SATA errors. 
Please check if you see matching errors in dmesg(1).



David

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/21/24 00:30, Max Nikulin wrote:

On 21/01/2024 03:23, gene heskett wrote:

On 1/20/24 10:24, Max Nikulin wrote:

On 19/01/2024 06:10, gene heskett wrote:
ID# ATTRIBUTE_NAME  FLAG VALUE WORST THRESH TYPE UPDATED 
WHEN_FAILED RAW_VALUE


190 Airflow_Temperature_Cel 0x0032   071   049   000    Old_age 
Always   -   29


Initial 100 decreased to 49 means that sometimes the drive is hot 
enough.


I've been under the impression that 100C was the absolute temp limit


Do not confuse normalized values (100 means shiny new, 0 means really 
old or damaged) and RAW_VALUE. For some drives smartctl -x may report 
history of temperature measurements, but I think summer values are 
already unavailable.


and it not been over 36C that I know of according to gkrellm which s 
set to monitor that stuff in real time. Right now nothing in the 
system is north of 32C, might get to 36C


71 <-> 29 °C and 49 <-> 36 °C mapping might be possible, but I would 
expect higher temperature for 49.


I read up on the manpage.
New -x version for this SSD attached

Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-17-rt-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 870 EVO 1TB
Serial Number:S626NF0R302498T
LU WWN Device Id: 5 002538 f413394a5
Firmware Version: SVT01B6Q
User Capacity:1,000,204,886,016 bytes [1.00 TB]
Sector Size:  512 bytes logical/physical
Rotation Rate:Solid State Device
Form Factor:  2.5 inches
TRIM Command: Available, deterministic, zeroed
Device is:In smartctl database 7.3/5319
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:Sun Jan 21 00:44:08 2024 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status:  ( 117) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline 
data collection:(0) seconds.
Offline data collection
capabilities:(0x53) SMART execute Offline immediate.
Auto Offline data collection on/off 
support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities:(0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability:(0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine 
recommended polling time:(   2) minutes.
Extended self-test routine
recommended polling time:(  85) minutes.
SCT capabilities:  (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAGSVALUE WORST THRESH FAIL RAW_VALUE
  5 Reallocated_Sector_Ct   PO--CK   094   094   010-64
  9 Power_On_Hours  -O--CK   095   095   000-21194
 12 Power_Cycle_Count   -O--CK   099   099   000-86
177 Wear_Leveling_Count PO--C-   099   099   000-23
179 Used_Rsvd_Blk_Cnt_Tot   PO--C-   094   094   010-64
181 Program_Fail_Cnt_Total  -O--CK   100   100   010-0
182 Erase_Fail_Count_Total  -O--CK   100   100   010-0
183 

Re: smartctl cannot access my storage, need syntax help

[Max Nikulin]

On 21/01/2024 03:23, gene heskett wrote:

On 1/20/24 10:24, Max Nikulin wrote:

On 19/01/2024 06:10, gene heskett wrote:
ID# ATTRIBUTE_NAME  FLAG VALUE WORST THRESH TYPE UPDATED  
WHEN_FAILED RAW_VALUE


190 Airflow_Temperature_Cel 0x0032   071   049   000    Old_age 
Always   -   29


Initial 100 decreased to 49 means that sometimes the drive is hot enough.


I've been under the impression that 100C was the absolute temp limit


Do not confuse normalized values (100 means shiny new, 0 means really 
old or damaged) and RAW_VALUE. For some drives smartctl -x may report 
history of temperature measurements, but I think summer values are 
already unavailable.


and it not been over 36C that I know of according to gkrellm which s set 
to monitor that stuff in real time. Right now nothing in the system is 
north of 32C, might get to 36C


71 <-> 29 °C and 49 <-> 36 °C mapping might be possible, but I would 
expect higher temperature for 49.


# 2  Extended offline    Completed: read failure   50% 
10917 1847474376
# 3  Extended offline    Completed: read failure   50% 
10586 1847474376


May it happen that disk firmware does not remap failed sectors to 
allow the user to identify what file is damaged?


IDK Max. I know the microware os9 file system well enough to connect the 
dots, but have little knowledge for how one might do this with ext4.


If you are motivated enough then docs either for badblocks or for some 
data recovery software may give you a recipe. A search engine should 
help to find it.

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/20/24 10:24, Max Nikulin wrote:

On 19/01/2024 06:10, gene heskett wrote:

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG VALUE WORST THRESH TYPE  
UPDATED  WHEN_FAILED RAW_VALUE
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   085   085   010    Pre-fail  
Always   -   168
183 Runtime_Bad_Block   0x0013   085   085   010    Pre-fail  
Always   -   168


85 is still far enough from 10, however the change is noticeable.

190 Airflow_Temperature_Cel 0x0032   071   049   000    Old_age   
Always   -   29


Initial 100 decreased to 49 means that sometimes the drive is hot 
enough.


I've been under the impression that 100C was the absolute temp limit, 
and it not been over 36C that I know of according to gkrellm which s set 
to monitor that stuff in real time. Right now nothing in the system is 
north of 32C, might get to 36C at the end of a 9 minute build of 
something in OpenSCAD.


 On the other hand the raw value of 29 is likely centigrade

degrees and it is not really hot for the normalized value of 71.


this is true, all reported temps are in C.


SMART Self-test log structure revision number 1
Num  Test_Description    Status  Remaining  
LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure   50% 
21128 1847474744
# 2  Extended offline    Completed: read failure   50% 
10917 1847474376
# 3  Extended offline    Completed: read failure   50% 
10586 1847474376


May it happen that disk firmware does not remap failed sectors to allow 
the user to identify what file is damaged?


IDK Max. I know the microware os9 file system well enough to connect the 
dots, but have little knowledge for how one might do this with ext4.


Thanks Max, take care & stay well.

Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[Max Nikulin]

On 19/01/2024 06:10, gene heskett wrote:

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG VALUE WORST THRESH TYPE  UPDATED  
WHEN_FAILED RAW_VALUE
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   085   085   010Pre-fail  Always   
-   168
183 Runtime_Bad_Block   0x0013   085   085   010Pre-fail  Always   
-   168


85 is still far enough from 10, however the change is noticeable.


190 Airflow_Temperature_Cel 0x0032   071   049   000Old_age   Always   
-   29


Initial 100 decreased to 49 means that sometimes the drive is hot 
enough. On the other hand the raw value of 29 is likely centigrade 
degrees and it is not really hot for the normalized value of 71.



SMART Self-test log structure revision number 1
Num  Test_DescriptionStatus  Remaining  LifeTime(hours)  
LBA_of_first_error
# 1  Extended offlineCompleted: read failure   50% 21128 
1847474744
# 2  Extended offlineCompleted: read failure   50% 10917 
1847474376
# 3  Extended offlineCompleted: read failure   50% 10586 
1847474376


May it happen that disk firmware does not remap failed sectors to allow 
the user to identify what file is damaged?

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/19/24 21:34, gene heskett wrote:

On 1/19/24 20:29, Felix Miata wrote:

gene heskett composed on 2024-01-19 19:09 (UTC-0500):

On 1/19/24 15:56, David Christensen wrote:

https://www.cablematters.com/pc-187-156-3-pack-straight-60-gbps-sata-iii-cable.aspx



Cheap enough at 18", ordered 4 packs of 3 for service & build stock,
thanks David.


Among the elements of that page, opened in web browser lacking JS 
support, was

absence of a price, and also were the following "features":

Serial ATA/150
and
Fast data transfer rate of up to 150 Mbps

Those describe SATA revision 1.0 (1.5 Gbit/s), not SATA revision 2.0 
(300MB/s, 3.0

Gbit/s), not SATA revision 3.0 (600MB/s, 6.0 Gbit/s).
https://en.wikipedia.org/wiki/SATA

With JS enabled, the page radically changed to show $8.49 for a 3-pack 
of 6.0

Gbit/s cables.


They had 2 lengths, 24" will if everything isn't good, sign on as 
sata-II but the 18" I bought claim sata-III.



I bought black cables, 18" and 24", straight-straight and straight-90. 
The older ones are labeled "Serial ATA 6G".  The newer ones are labeled 
"Serial ATA3.2".



David

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/19/24 20:29, Felix Miata wrote:

gene heskett composed on 2024-01-19 19:09 (UTC-0500):


On 1/19/24 15:56, David Christensen wrote:



No sign of that snipped stuff.


https://www.cablematters.com/pc-187-156-3-pack-straight-60-gbps-sata-iii-cable.aspx



Cheap enough at 18", ordered 4 packs of 3 for service & build stock,
thanks David.


Among the elements of that page, opened in web browser lacking JS support, was
absence of a price, and also were the following "features":

Serial ATA/150
and
Fast data transfer rate of up to 150 Mbps

Those describe SATA revision 1.0 (1.5 Gbit/s), not SATA revision 2.0 (300MB/s, 
3.0
Gbit/s), not SATA revision 3.0 (600MB/s, 6.0 Gbit/s).
https://en.wikipedia.org/wiki/SATA

With JS enabled, the page radically changed to show $8.49 for a 3-pack of 6.0
Gbit/s cables.


They had 2 lengths, 24" will if everything isn't good, sign on as 
sata-II but the 18" I bought claim sata-III.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[Felix Miata]

gene heskett composed on 2024-01-19 19:09 (UTC-0500):

> On 1/19/24 15:56, David Christensen wrote:

> No sign of that snipped stuff.
> 
>> https://www.cablematters.com/pc-187-156-3-pack-straight-60-gbps-sata-iii-cable.aspx

> Cheap enough at 18", ordered 4 packs of 3 for service & build stock, 
> thanks David. 

Among the elements of that page, opened in web browser lacking JS support, was
absence of a price, and also were the following "features":

Serial ATA/150
and
Fast data transfer rate of up to 150 Mbps

Those describe SATA revision 1.0 (1.5 Gbit/s), not SATA revision 2.0 (300MB/s, 
3.0
Gbit/s), not SATA revision 3.0 (600MB/s, 6.0 Gbit/s).
https://en.wikipedia.org/wiki/SATA

With JS enabled, the page radically changed to show $8.49 for a 3-pack of 6.0
Gbit/s cables.
-- 
Evolution as taught in public schools is, like religion,
based on faith, not based on science.

 Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/19/24 15:56, David Christensen wrote:
No sign of that snipped stuff.


https://www.cablematters.com/pc-187-156-3-pack-straight-60-gbps-sata-iii-cable.aspx


Cheap enough at 18", ordered 4 packs of 3 for service & build stock, 
thanks David.



I call that the "wiggle" test.


So do I but I've had to explain it.  Several times.

Now they'll have to dig me out, got around 16" of white stuff in the 
last 36 hrs. I believe winter has arrived.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/18/24 23:23, gene heskett wrote:

On 1/19/24 00:55, David Christensen wrote:
I am unclear if those errors are inside the SSD or if they are the 
SATA communications link between the SSD and the motherbaord or HBA 
port and/or main memory (?).  Does dmesg(1) show anything?


I'm not sure what I should be looking for, and I don't see anything that 
is looping to correct an error.  Suggested grep targets?



Here is a dmesg(1) excerpt from 2014 -- Debian 7, good SSD, bad SATA cable:

[2.086360] ata3.00: ATA-9: INTEL SSDSC2CW060A3, 400i, max UDMA/133
[2.086365] ata3.00: 117231408 sectors, multi 16: LBA48 NCQ (depth 
31/32), AA

[2.096265] ata3.00: configured for UDMA/133
[   14.718054] EXT4-fs (dm-0): mounted filesystem with ordered data 
mode. Opts: (null)
[   18.449227] EXT4-fs (sda1): mounted filesystem with ordered data 
mode. Opts: (null)
[   20.157693] ata3.00: exception Emask 0x10 SAct 0x40 SErr 0xc1 
action 0x6 frozen

[   20.157699] ata3.00: irq_stat 0x0800, interface fatal error
[   20.157703] ata3: SError: { RecovData Handshk LinkSeq }
[   20.157709] ata3.00: failed command: WRITE FPDMA QUEUED
[   20.157716] ata3.00: cmd 61/08:b0:a0:e0:61/00:00:00:00:00/40 tag 22 
ncq 4096 out

[   20.157721] ata3.00: status: { DRDY }
[   20.157727] ata3: hard resetting link
[   20.473489] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   20.484835] ata3.00: configured for UDMA/133
[   20.484847] ata3: EH complete
[   21.059825] ata3.00: exception Emask 0x10 SAct 0x4000 SErr 0x400100 
action 0x6 frozen

[   21.059831] ata3.00: irq_stat 0x0800, interface fatal error
[   21.059835] ata3: SError: { UnrecovData Handshk }
[   21.059840] ata3.00: failed command: WRITE FPDMA QUEUED
[   21.059848] ata3.00: cmd 61/08:70:50:e2:61/00:00:00:00:00/40 tag 14 
ncq 4096 out

[   21.059853] ata3.00: status: { DRDY }
[   21.059859] ata3: hard resetting link
[   21.376135] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   21.397234] ata3.00: configured for UDMA/133
[   21.397246] ata3: EH complete
[   22.590805] ata3.00: exception Emask 0x10 SAct 0x600 SErr 0x400100 
action 0x6 frozen

[   22.590811] ata3.00: irq_stat 0x0800, interface fatal error
[   22.590815] ata3: SError: { UnrecovData Handshk }
[   22.590819] ata3.00: failed command: WRITE FPDMA QUEUED
[   22.590826] ata3.00: cmd 61/08:48:f0:ee:1d/00:00:00:00:00/40 tag 9 
ncq 4096 out

[   22.590831] ata3.00: status: { DRDY }
[   22.590834] ata3.00: failed command: WRITE FPDMA QUEUED
[   22.590840] ata3.00: cmd 61/08:50:70:ef:1d/00:00:00:00:00/40 tag 10 
ncq 4096 out

[   22.590844] ata3.00: status: { DRDY }
[   22.590851] ata3: hard resetting link
[   22.909955] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   22.921525] ata3.00: configured for UDMA/133
[   22.937878] ata3: EH complete
[   22.938635] ata3: limiting SATA link speed to 3.0 Gbps
[   22.938638] ata3.00: exception Emask 0x10 SAct 0x40 SErr 0x400100 
action 0x6 frozen

[   22.938640] ata3.00: irq_stat 0x0800, interface fatal error
[   22.938642] ata3: SError: { UnrecovData Handshk }
[   22.938645] ata3.00: failed command: WRITE FPDMA QUEUED
[   22.938648] ata3.00: cmd 61/60:b0:20:28:66/00:00:00:00:00/40 tag 22 
ncq 49152 out

[   22.938650] ata3.00: status: { DRDY }
[   22.938652] ata3: hard resetting link
[   23.257418] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[   23.269251] ata3.00: configured for UDMA/133
[   23.285387] ata3: EH complete


In any case, make sure that you are using SATA III 6 Gbps cables with 
locking connectors for your drives and that all the connections are good.


That's hard to verify once the cables are removed from the packing. all 
are black, with locking clips  There is a cable maker under every tree 
in china so I'n not swearing any are up to specs, I've had cable problem 
in the past but usually a magenta colored on that is over 2 years old, 
If you have a known good src on straight on cables, please share.  You 
would be doing everyone a favor. 



https://www.cablematters.com/pc-187-156-3-pack-straight-60-gbps-sata-iii-cable.aspx

https://www.cablematters.com/pc-188-156-cable-matters-3-pack-90-degree-right-angle-60-gbps-sata-iii-cable-18-inches.aspx


Test what you have by taking a wooden stick and moving each one a 
centimeter or so, if the log blows up with sata resets, bingo, bad 
cable. replace it asap.



I call that the "wiggle" test.


David

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/19/24 00:03, Anssi Saari wrote:

My only mdraid was on raw partitions but that never had any issues. I
think zfs effectively does the same, no partitions.



You can do it either way on ZFS.


David

Re: To partition or not to partition MD arrays (Was Re: smartctl cannotaccess my storage, need syntax help)

[Franco Martelli]

On 19/01/24 at 20:14, Nicolas George wrote:

Franco Martelli (12024-01-19):

One case against using partitions on mdraid: if your array gets messed
up, you get to recreate those partition tables yourself and that's just
hilarious if you don't have a backup. Happened to a friend of mine,
reason was a UPS brownout.

How can I get a backup of mdadm RAID partition?


You do not need a backup of the RAID partitions, that would be terribly
inefficient. You need a backup of the partition table.


Yes, I agree of course. I was asking this to Anssi because it looks like 
strange to me to have the backup of the partitions, as he pointed (for 
my understanding)




Which, if you are organized, you already have in
$notes_dir/$hostname/install.md as something that looks like this:

```
sudo sfdisk /dev/sdX <

The partitions table of my HDD is part of my backup.

Cheers,

--
Franco Martelli

Re: To partition or not to partition MD arrays (Was Re: smartctl cannotaccess my storage, need syntax help)

[Nicolas George]

Franco Martelli (12024-01-19):
> > One case against using partitions on mdraid: if your array gets messed
> > up, you get to recreate those partition tables yourself and that's just
> > hilarious if you don't have a backup. Happened to a friend of mine,
> > reason was a UPS brownout.
> How can I get a backup of mdadm RAID partition?

You do not need a backup of the RAID partitions, that would be terribly
inefficient. You need a backup of the partition table.

Which, if you are organized, you already have in
$notes_dir/$hostname/install.md as something that looks like this:

```
sudo sfdisk /dev/sdX <

signature.asc
Description: PGP signature

Re: To partition or not to partition MD arrays (Was Re: smartctl cannotaccess my storage, need syntax help)

[Franco Martelli]

On 19/01/24 at 09:03, Anssi Saari wrote:

One case against using partitions on mdraid: if your array gets messed
up, you get to recreate those partition tables yourself and that's just
hilarious if you don't have a backup. Happened to a friend of mine,
reason was a UPS brownout.


How can I get a backup of mdadm RAID partition? And which tool to backup 
the whole disks of an array? The only tool that it comes in mind it is 
"dd" that it isn't a viable solution for me.
I think is useless to backup the raw data stored in a partition or the 
whole disk. I backup files and directories stored in the filesystem not 
raw data. If an error occurs in the RAID, mdadm takes care to warn me 
via email... I hope!



I think he scanned his disks for copies of
the superblock but didn't find any and then somehow with a lot of hassle
eventually figured out what the partition tables were.

So in a catastrophe, partition tables are one more obstacle to cross
before you can start actually recovering your data.


Me too ran into a catastrophe scenario, I had lost /dev/md0, the reason 
was using hibernate (suspend to disk) in a logical volume placed inside 
the RAID. I think it was damaged the RAID metadata.
I got rid of this using Debian-installer, I thought that I had loosed 
everything and I prepared for reinstall, when Debian-installer asked me 
to create the new RAID I specify all the four partitions, I saved, and 
magically the logical device and all my logical volumes, embedded in the 
old RAID, reappeared. To partition was not a trouble in those circumstances.




My only mdraid was on raw partitions but that never had any issues. I
think zfs effectively does the same, no partitions.


Which raw partitions? Maybe did you mean without partitions? I never 
used zfs it's full featured, I prefer to keep the things simple: RAID -> 
LVM -> ext4


Cheers,
--
Franco Martelli

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/19/24 04:50, Thomas Schmitt wrote:

Hi,

Anssi Saari

It does seem strange to me, even in MS-DOS era I was able to set a
terminal scrollback to 5000 lines without issue, when RAM was maybe 4 MB
and a DOS terminal program probably had access to way less than that.


I have no problems with 130 xterms of 10,000 lines each.



So does rsync really generate gigabytes of verbose output?


rsync can be extremely verbose when the number of transferred files is
very high.



Or is xfce-terminal storing the scrollback in a very inefficient way?


I would not be astonished to learn that the luxury ornamented terminals
of the various desktops waste many extra bytes when memorizing plain text.
But the real bug is the fact that the scroll back memory is unlimited and
can summon the OOM killer. (I imagine it like the Discworld Death of Rats.)

If i were a user of Xfce i would report this as bug to its Debian
maintainer. Bug title "xfce-terminal: A landmine on the kids' playground".


Have a nice day :)

Thomas

.

Excellent description Thomas. Love it.
Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/19/24 03:12, Anssi Saari wrote:

gene heskett  writes:


The OOM death of the system was the xfce4 terminal apparently being
set for unlimited scrollback and that was eating the memory. Switching
to Konsole with has the ability to control the scrollback to 200
lines, and its taken all 32G's as .cache and 1536 1k blocks of swap,
and its working w/o any OOM actions I've detected.


It does seem strange to me, even in MS-DOS era I was able to set a
terminal scrollback to 5000 lines without issue, when RAM was maybe 4 MB
and a DOS terminal program probably had access to way less than that. So
does rsync really generate gigabytes of verbose output? Or is
xfce-terminal storing the scrollback in a very inefficient way?

That I can't answer, other than -v outputs a full from / pathlist to 
everyfile it touchs, and if storing that in ram, I can sure see it 
eating 32G very quickly when it is moving 335G, it only got around 13G 
moved before OOM struck and killed the system, on each of probably 15 
attempts. Knowing that the tech of an SSD and the common micro-sd has a 
relatively limited actual write speed after in has used up its input 
cache of fast ram, I took the v off the -av, and them limited it to 
10megs a second, it took around 9 hours and the system acted normally, 
no OOM problens. I have edited the /etc/stab and am now running on that 
copy for /home.  The raid is now automounted to /raid10 and says its 
valid, despite the 4th drives log being a mess. My thoughts are to 
reverse the copy and put it in crontab to keep an uptodate backup of 
/home until I can re-invent my wrappers for amanda. /home is by far the 
biggest glop of data, and none of my printers or cnc machines will use 
more that 10G reach, so I'm inclinded to think of the other 8T of drives 
as an lvm managed 8T, which should give me room enough to keep 30 days 
worth of amanda's way of doing things.


But I'm hibernating for the nonce, I woke up at 6 with 6" of new snow on 
the deck, and the weather fabricators are promising another 24 hours of 
that, might wind up with 3 or 4 feet of it. I've got coffee, the 
freezers are well stocked.  Boring but safe.


All the messy logs were at hour 21027 so that was a single actual event, 
probably caused by OOM.


Take care Anssi, stay warm, dry and well where ever you are.

Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

Anssi Saari
> It does seem strange to me, even in MS-DOS era I was able to set a
> terminal scrollback to 5000 lines without issue, when RAM was maybe 4 MB
> and a DOS terminal program probably had access to way less than that.

I have no problems with 130 xterms of 10,000 lines each.


> So does rsync really generate gigabytes of verbose output?

rsync can be extremely verbose when the number of transferred files is
very high.


> Or is xfce-terminal storing the scrollback in a very inefficient way?

I would not be astonished to learn that the luxury ornamented terminals
of the various desktops waste many extra bytes when memorizing plain text.
But the real bug is the fact that the scroll back memory is unlimited and
can summon the OOM killer. (I imagine it like the Discworld Death of Rats.)

If i were a user of Xfce i would report this as bug to its Debian
maintainer. Bug title "xfce-terminal: A landmine on the kids' playground".


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[Anssi Saari]

gene heskett  writes:

> The OOM death of the system was the xfce4 terminal apparently being
> set for unlimited scrollback and that was eating the memory. Switching
> to Konsole with has the ability to control the scrollback to 200
> lines, and its taken all 32G's as .cache and 1536 1k blocks of swap,
> and its working w/o any OOM actions I've detected.

It does seem strange to me, even in MS-DOS era I was able to set a
terminal scrollback to 5000 lines without issue, when RAM was maybe 4 MB
and a DOS terminal program probably had access to way less than that. So
does rsync really generate gigabytes of verbose output? Or is
xfce-terminal storing the scrollback in a very inefficient way?

Re: smartctl cannot access my storage, need syntax help

[Anssi Saari]

Franco Martelli  writes:

> I don't know if it is a good idea, in fact it exists a special
> partition type for RAID array listed in fdisk, I used that for my
> RAID:

One case against using partitions on mdraid: if your array gets messed
up, you get to recreate those partition tables yourself and that's just
hilarious if you don't have a backup. Happened to a friend of mine,
reason was a UPS brownout. I think he scanned his disks for copies of
the superblock but didn't find any and then somehow with a lot of hassle
eventually figured out what the partition tables were.

So in a catastrophe, partition tables are one more obstacle to cross
before you can start actually recovering your data.

My only mdraid was on raw partitions but that never had any issues. I
think zfs effectively does the same, no partitions.

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/19/24 00:55, David Christensen wrote:

On 1/18/24 15:10, gene heskett wrote:

On 1/18/24 16:08, David Christensen wrote:

On 1/18/24 03:47, gene heskett wrote:
I have issued a smartctl -tlong on all 4 drives, results in about 3 
hours.



A SMART long test should find and fix any read errors.

Which has now been done on all 4 SSD. but the log is still a mess. 4th 
one in particular, smartctl -a /dev/sdg attached.



179 Used_Rsvd_Blk_Cnt_Tot   0x0013   085   085   010    Pre-fail  Always 
   -   168


183 Runtime_Bad_Block   0x0013   085   085   010    Pre-fail  Always 
   -   168
187 Uncorrectable_Error_Cnt 0x0032   099   099   000    Old_age   Always 
   -   3275


195 ECC_Error_Rate  0x001a   199   199   000    Old_age   Always 
   -   3275


Error 3332 occurred at disk power-on lifetime: 21027 hours (876 days + 3 
hours)
   When the command that caused the error occurred, the device was 
active or idle.


   After command completion occurred, registers were:
   ER ST SC SN CL CH DH
   -- -- -- -- -- -- --
   40 51 38 e8 ea 67 40  Error: WP at LBA = 0x0067eae8 = 6810344

   Commands leading to the command that caused the error were:
   CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
   -- -- -- -- -- -- -- --    
   61 18 38 e8 ea 67 40 07  15:17:03.046  WRITE FPDMA QUEUED
   60 00 30 00 5e a9 40 06  15:17:03.046  READ FPDMA QUEUED
   60 28 28 00 f4 87 40 05  15:17:03.046  READ FPDMA QUEUED
   60 00 20 00 7c a9 40 04  15:17:03.046  READ FPDMA QUEUED
   60 00 18 00 4a a9 40 03  15:17:03.046  READ FPDMA QUEUED

Error 3331 occurred at disk power-on lifetime: 21027 hours (876 days + 3 
hours)


Error 3330 occurred at disk power-on lifetime: 21027 hours (876 days + 3 
hours)


Error 3329 occurred at disk power-on lifetime: 21027 hours (876 days + 3 
hours)


Error 3328 occurred at disk power-on lifetime: 21027 hours (876 days + 3 
hours)



I am unclear if those errors are inside the SSD or if they are the SATA 
communications link between the SSD and the motherbaord or HBA port 
and/or main memory (?).  Does dmesg(1) show anything?


I'm not sure what I should be looking for, and I don't see anything that 
is looping to correct an error.  Suggested grep targets?


In any case, make sure that you are using SATA III 6 Gbps cables with 
locking connectors for your drives and that all the connections are good.


That's hard to verify once the cables are removed from the packing. all 
are black, with locking clips  There is a cable maker under every tree 
in china so I'n not swearing any are up to specs, I've had cable problem 
in the past but usually a magenta colored on that is over 2 years old, 
If you have a known good src on straight on cables, please share.  You 
would be doing everyone a favor. No hot red need apply. People think its 
pretty, but the die that gives the color, eats the copper in the cable.


I am the src of the internet legend about that, first observed in the 
early 1970's when all the cb radio mic cables switched from dull red to 
this bright red/magemta as the tx wire in multiconductor cables. And 
that wire literally dissolved the copper in the hot red conductor to a 
dull rusty powder in 2 years.
And its been doing that same failure in sata cables of that color for a 
decade now.


Test what you have by taking a wooden stick and moving each one a 
centimeter or so, if the log blows up with sata resets, bingo, bad 
cable. replace it asap.



When deploying an SSD into a new role, I like to do a "secure erase" 
followed by a SMART long test.


not fam with that, I usually just reformat.  But I'll not do that 
until I have amanda running again.



Secure erase will erase all of the blocks in the drive, including those 
that are held in reserve.  This both verifies that each block can be 
erased, and provides maximum performance what you put the disk into 
service and start writing to it.




Thanks David, take care & stay well


Likewise.  :-)


David


.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/18/24 15:10, gene heskett wrote:

On 1/18/24 16:08, David Christensen wrote:

On 1/18/24 03:47, gene heskett wrote:
I have issued a smartctl -tlong on all 4 drives, results in about 3 
hours.



A SMART long test should find and fix any read errors.

Which has now been done on all 4 SSD. but the log is still a mess. 4th 
one in particular, smartctl -a /dev/sdg attached.



179 Used_Rsvd_Blk_Cnt_Tot   0x0013   085   085   010Pre-fail  Always 
  -   168


183 Runtime_Bad_Block   0x0013   085   085   010Pre-fail  Always 
  -   168
187 Uncorrectable_Error_Cnt 0x0032   099   099   000Old_age   Always 
  -   3275


195 ECC_Error_Rate  0x001a   199   199   000Old_age   Always 
  -   3275


Error 3332 occurred at disk power-on lifetime: 21027 hours (876 days + 3 
hours)
  When the command that caused the error occurred, the device was 
active or idle.


  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 38 e8 ea 67 40  Error: WP at LBA = 0x0067eae8 = 6810344

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --    
  61 18 38 e8 ea 67 40 07  15:17:03.046  WRITE FPDMA QUEUED
  60 00 30 00 5e a9 40 06  15:17:03.046  READ FPDMA QUEUED
  60 28 28 00 f4 87 40 05  15:17:03.046  READ FPDMA QUEUED
  60 00 20 00 7c a9 40 04  15:17:03.046  READ FPDMA QUEUED
  60 00 18 00 4a a9 40 03  15:17:03.046  READ FPDMA QUEUED

Error 3331 occurred at disk power-on lifetime: 21027 hours (876 days + 3 
hours)


Error 3330 occurred at disk power-on lifetime: 21027 hours (876 days + 3 
hours)


Error 3329 occurred at disk power-on lifetime: 21027 hours (876 days + 3 
hours)


Error 3328 occurred at disk power-on lifetime: 21027 hours (876 days + 3 
hours)



I am unclear if those errors are inside the SSD or if they are the SATA 
communications link between the SSD and the motherbaord or HBA port 
and/or main memory (?).  Does dmesg(1) show anything?



In any case, make sure that you are using SATA III 6 Gbps cables with 
locking connectors for your drives and that all the connections are good.



When deploying an SSD into a new role, I like to do a "secure erase" 
followed by a SMART long test.


not fam with that, I usually just reformat.  But I'll not do that until 
I have amanda running again.



Secure erase will erase all of the blocks in the drive, including those 
that are held in reserve.  This both verifies that each block can be 
erased, and provides maximum performance what you put the disk into 
service and start writing to it.




Thanks David, take care & stay well


Likewise.  :-)


David

Re: smartctl cannot access my storage, need syntax help

[David Wright]

On Thu 18 Jan 2024 at 00:57:07 (-0800), David Christensen wrote:
> On 1/17/24 22:44, gene heskett wrote:
> > One thing that bothers me is there is no way the installers parted
> > shows partition names for non-raid disks. To me that is a serious
> > bug. It appears from the help that it can LABEL a partition but
> > can't read that LABEL.
> 
> When installing to UEFI/GPT, I am able to label partitions in the
> Debian Installer, the labels are visible in the installer, and the
> labels persist on disk after installation is complete.

Agreed, and that doesn't depend on UEFI; MBR/GPT disks show
the same behaviour. But those are PARTLABELS.

But it may be that Gene meant filesystem LABELs.

Gene, to check/display the LABELs, just place, in turn, the
highlight on the line for each partition, like:

  │   >   #5   31.5 GBext4Viva-B▒ │

press Return for it to display:

  │ Partition settings: │   
  │ │   
  │Name:  Viva-B│   
  │Use as:Ext4 journaling file system   │   
  │ │   
  │Format the partition:  yes, format it│   
  │Mount point:   / │   
  │Mount options: defaults  │   
  │Label: viva05 ←← │   
  │Reserved blocks:   5%│   
  │ │   
  │Done setting up the partition│   

where Name: ⇒ PARTLABEL and Label: ⇒ LABEL.

Then select "Done setting up …" or  to back out each time.

Cheers,
David.

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/18/24 16:08, David Christensen wrote:

On 1/18/24 03:47, gene heskett wrote:

On 1/18/24 03:57, David Christensen wrote:
The old /home RAID10 still has its metadata on disk.  I would install 
the "mdadm" package, edit /etc/fstab, copy and rework the old /home 
line (new mount point, add option "ro"), create the mount point, and 
mount.


I believe mdadm is already installed. At least enough to collect and 
mount this raid10 and use it for /home for the last nearly 2 years.



I made the suggestion to install the "mdadm" package because I thought 
you were going to do a fresh install of Debian.



Now after all this folderall, all 4 of the SSD's are reporting read 
errors at very high lba's.


all 4 drives are reporting the same poh, 21027 hours for the occurence 
of the error, that sounds like it could be just one crash or dirty 
power down.  In which case it s/b repairable


  Do we have a repair utility that will force the drive to reallocate 
a spare sector and fix those?
I have issued a smartctl -tlong on all 4 drives, results in about 3 
hours.



A SMART long test should find and fix any read errors.

Which has now been done on all 4 SSD. but the log is still a mess. 4th 
one in particular, smartctl -a /dev/sdg attached.


When deploying an SSD into a new role, I like to do a "secure erase" 
followed by a SMART long test.
not fam with that, I usually just reformat.  But I'll not do that until 
I have amanda running again.


Thanks David, take care & stay well


David




.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-17-rt-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 870 EVO 1TB
Serial Number:S626NF0R302509W
LU WWN Device Id: 5 002538 f413394b0
Firmware Version: SVT01B6Q
User Capacity:1,000,204,886,016 bytes [1.00 TB]
Sector Size:  512 bytes logical/physical
Rotation Rate:Solid State Device
Form Factor:  2.5 inches
TRIM Command: Available, deterministic, zeroed
Device is:In smartctl database 7.3/5319
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:Thu Jan 18 18:02:48 2024 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status:  ( 117) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline 
data collection:(0) seconds.
Offline data collection
capabilities:(0x53) SMART execute Offline immediate.
Auto Offline data collection on/off 
support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities:(0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability:(0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine 
recommended polling time:(   2) minutes.
Extended self-test routine
recommended polling time:(  85) minutes.
SCT capabilities:  (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG VALUE WORST THRESH TYPE  UPDATED  
WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   085   085   010Pre-fail  Always   
-   168
  9 Power_On_Hours  0x0032   095   095   000Old_age   Always   
-   21139
 12 Power_Cycle_Count   

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/18/24 03:47, gene heskett wrote:

On 1/18/24 03:57, David Christensen wrote:
The old /home RAID10 still has its metadata on disk.  I would install 
the "mdadm" package, edit /etc/fstab, copy and rework the old /home 
line (new mount point, add option "ro"), create the mount point, and 
mount.


I believe mdadm is already installed. At least enough to collect and 
mount this raid10 and use it for /home for the last nearly 2 years.



I made the suggestion to install the "mdadm" package because I thought 
you were going to do a fresh install of Debian.



Now after all this folderall, all 4 of the SSD's are reporting read 
errors at very high lba's.


all 4 drives are reporting the same poh, 21027 hours for the occurence 
of the error, that sounds like it could be just one crash or dirty power 
down.  In which case it s/b repairable


  Do we have a repair utility that will force the drive to reallocate a 
spare sector and fix those?

I have issued a smartctl -tlong on all 4 drives, results in about 3 hours.



A SMART long test should find and fix any read errors.


When deploying an SSD into a new role, I like to do a "secure erase" 
followed by a SMART long test.



David

Re: To partition or not to partition MD arrays (Was Re: smartctl cannotaccess my storage, need syntax help)

[Andy Smith]

Hi,

On Thu, Jan 18, 2024 at 10:28:30AM -0600, Nicholas Geovanis wrote:
> Sounds like this group has finally achieved a long overdue consensus. How
> many times since LVM was ready for root/boot volumes have I been told that
> using partitions was necessary good practice. Even had that in job
> interviews, where half the team would grin at me saying it and the other
> half scowling at my "poor practice".
> 
> Now we know it was just personal preference all along. Like somebody said
> :-)

Look, if you're going to resolve this thread so quickly all it means
is that someone is going to have to mention home.arpa or their time
zone setting again. We have strict quotas here for the amount of
circular repeating "you don't do things like me therefore you are
wrong and here are a selection of Internet standards to back me up"
threads that must be taking place at once.



Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: smartctl cannot access my storage, need syntax help

[Curt]

On 2024-01-17, Thomas Schmitt  wrote:
> Hi,
>
> Curt wrote:
>> I discovered a couple of discussions of the phenomenon, the upshot of which
>> were:
>> 1) That's what you get when you purchase cheap SSDs.
>> https://www.reddit.com/r/truenas/comments/s0rrpo/two_sata_ssds_with_identical_serial_numbers/
>> 2) SSDs belonging to the same software RAID show identical serial numbers
>> in software, but these numbers don't match the serial numbers printed on the
>> SSDs themselves.
>> https://www.reddit.com/r/truenas/comments/s0rrpo/two_sata_ssds_with_identical_serial_numbers/
>
> Those URLs are identical. (OMG ! Is it contageous ?)

Human error may very be:
https://www.reddit.com/r/synology/comments/18fe6ez/how_to_fix_2_drives_with_same_serial_number/


> Number 2 would match my suspicion that some layer in the disk driving
> gets confused and mixes up the serial numbers.
>
>
>> But you said *similar*.
>
> By "colliding serial numbers" i mean indeed "identical serial numbers".
>
> How cheap the disks may ever be, that would be no excuse for not making
> them individually distinguishable.
>
>
>> As Gene's threads have too many movable parts
>> for me to follow, on that point I couldn't say.
>
> This one begins to gain presence in the web. So one can use search engines
> and AI to untangle its sub-threads. I meanwhile participate in two of them:
> serial number collision, rsync caused OOM killer (solved now, but how ?).
>
>
> Have a nice day :)
>
> Thomas
>
>


-- 

normally start new xterms [was: Re: smartctl cannot access my storage, need syntax help]

[Max Nikulin]

On 18/01/2024 04:20, Thomas Schmitt wrote:


I normally start new xterms by

   xterm -ls -geometry 80x24 -bg wheat -fg black -sl 1 +sb &


Options may be put into ~/.Xresources

xterm*vt100.saveLines: 1
xterm*VT100.background: wheat
xterm*VT100.foreground: black
! etc

Use xrdb to merge changes without restarting X session. It is possible 
to have several presets (-name or -class), see /etc/X11/app-defaults/

Re: To partition or not to partition MD arrays (Was Re: smartctl cannotaccess my storage, need syntax help)

[Nicholas Geovanis]

On Wed, Jan 17, 2024, 9:35 PM gene heskett  wrote:

> On 1/17/24 19:54, Steve McIntyre wrote:
> > Andy Smith wrote:
> ...
> >> Then there will just be people going by taste.
> >>
> >> Personally I still put them directly on drives. If I ever get taken
> >> out by one of those crappy motherboards, I reserve the right to get
> >> a different religion. 
> >
> > I'm clearly a member of a third group of people,,, :-)
> >
> > Putting partitions on the RAID drives helps *me* identify them.
> >
> you aren't alone Steve.
> Cheers, Gene Heskett.
>

Sounds like this group has finally achieved a long overdue consensus. How
many times since LVM was ready for root/boot volumes have I been told that
using partitions was necessary good practice. Even had that in job
interviews, where half the team would grin at me saying it and the other
half scowling at my "poor practice".

Now we know it was just personal preference all along. Like somebody said
:-)

>

Re: To partition or not to partition MD arrays (Was Re: smartctl cannot access my storage, need syntax help)

[Steve McIntyre]

Hey Andy.

Andy Smith wrote:
>
>On Thu, Jan 18, 2024 at 12:53:43AM +, Steve McIntyre wrote:
>> I'm clearly a member of a third group of people,,, :-)
>
>Oh, I didn't mean to imply that those going by taste were in a
>minority! Taste, or possibly, "just never thought about it" could
>well be the biggest group. I was only talking about my observations
>of those who seem to hold strong opinions on this, usually to the
>point where they will advocate "their way" to others.

ACK!

>> Putting partitions on the RAID drives helps *me* identify them.
>
>So, I don't care what people do and I'm not trying to change your
>mind. Would you mind going into what makes "sda1" more identifiable
>for you than "sda" though?
>
>Or is it that you make use of partition labels for some extra info?

If I'm looking at disks on a system, the first thing I'll look for is
the partition table. If a disk has a partition table with "Linux RAID"
partitions viaible, that gives me a strong hint of what I should
expect on the disk. Especially if I'm swappings disk around between
systems, commisioning new systems and re-using disks etc.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
Can't keep my eyes from the circling sky,
Tongue-tied & twisted, Just an earth-bound misfit, I...

Re: To partition or not to partition MD arrays (Was Re: smartctl cannot access my storage, need syntax help)

[Andy Smith]

Hello,

On Thu, Jan 18, 2024 at 12:53:43AM +, Steve McIntyre wrote:
> I'm clearly a member of a third group of people,,, :-)

Oh, I didn't mean to imply that those going by taste were in a
minority! Taste, or possibly, "just never thought about it" could
well be the biggest group. I was only talking about my observations
of those who seem to hold strong opinions on this, usually to the
point where they will advocate "their way" to others.

> Putting partitions on the RAID drives helps *me* identify them.

So, I don't care what people do and I'm not trying to change your
mind. Would you mind going into what makes "sda1" more identifiable
for you than "sda" though?

Or is it that you make use of partition labels for some extra info?

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/18/24 03:57, David Christensen wrote:
On 1/17/24 22:44, gene heskett wrote:>> On 1/18/24 00:50, David 
Christensen wrote:
The migration took two passes because udev can't make up its alleged 
mind so I was finally forced to use the rescue mode to edit fstab to 
mount it by UUID and that worked, I've got /home on the copy right now.



Congratulations!  :-)


and I took the 60 G's of swap out too since I've never used more the 
20G with any gfx program, so I figure 47G's on /dev/sda is enough. 



1 GB swap works for me.  When a memory leak gets out of control, I do 
not have to wait long for the lock up.



So now none of the raid is mounted, but the 30+ second lag when 
opening a write path is still there, so I was erroneously blaming the 
raid. So I've narrowed the problem 



Good to know.


but w/o a good clue what to do next. 



Find the needle in the haystack or do a fresh install.  I prefer the 
latter, because I can estimate the effort and I am reasonably confident 
of the outcome.



One thing that bothers me is there is no way the installers parted 
shows partition names for non-raid disks. To me that is a serious bug. 
It appears from the help that it can LABEL a partition but can't read 
that LABEL.



When installing to UEFI/GPT, I am able to label partitions in the Debian 
Installer, the labels are visible in the installer, and the labels 
persist on disk after installation is complete.



parted when asked to print all does that just fine, but the | doesn't 
put it to less, so it scrolls off screen the top 60% of a parted's 
print all output at some fraction of C speed. Not exactly helpful. I 
have other things to do while I cogitate on what to do next.



The following works as expected on my machine:

2024-01-18 00:34:41 root@laalaa ~
# parted -l | less



Many thanks to all that helped.



YW.  :-)


If you use rsync(1), I suggest using some kind of integrity checking 
tool to verify that the source and destination file systems are 
identical.  I prefer BSD mtree(8):


I assume I'd have to remount the raid like to /raid?
Whew!  That's got more arguments than rsync...



The old /home RAID10 still has its metadata on disk.  I would install 
the "mdadm" package, edit /etc/fstab, copy and rework the old /home line 
(new mount point, add option "ro"), create the mount point, and mount.


I believe mdadm is already installed. At least enough to collect and 
mount this raid10 and use it for /home for the last nearly 2 years.
Now after all this folderall, all 4 of the SSD's are reporting read 
errors at very high lba's.


all 4 drives are reporting the same poh, 21027 hours for the occurence 
of the error, that sounds like it could be just one crash or dirty power 
down.  In which case it s/b repairable


 Do we have a repair utility that will force the drive to reallocate a 
spare sector and fix those?

I have issued a smartctl -tlong on all 4 drives, results in about 3 hours.


David

.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/17/24 22:44, gene heskett wrote:>> On 1/18/24 00:50, David 
Christensen wrote:
The migration took two passes because udev can't make up its alleged 
mind so I was finally forced to use the rescue mode to edit fstab to 
mount it by UUID and that worked, I've got /home on the copy right now.



Congratulations!  :-)


and I took the 60 G's of swap out too since I've never used more the 20G 
with any gfx program, so I figure 47G's on /dev/sda is enough. 



1 GB swap works for me.  When a memory leak gets out of control, I do 
not have to wait long for the lock up.



So now 
none of the raid is mounted, but the 30+ second lag when opening a write 
path is still there, so I was erroneously blaming the raid. So I've 
narrowed the problem 



Good to know.


but w/o a good clue what to do next. 



Find the needle in the haystack or do a fresh install.  I prefer the 
latter, because I can estimate the effort and I am reasonably confident 
of the outcome.



One thing that 
bothers me is there is no way the installers parted shows partition 
names for non-raid disks. To me that is a serious bug. It appears from 
the help that it can LABEL a partition but can't read that LABEL.



When installing to UEFI/GPT, I am able to label partitions in the Debian 
Installer, the labels are visible in the installer, and the labels 
persist on disk after installation is complete.



parted 
when asked to print all does that just fine, but the | doesn't put it to 
less, so it scrolls off screen the top 60% of a parted's print all 
output at some fraction of C speed. Not exactly helpful. I have other 
things to do while I cogitate on what to do next.



The following works as expected on my machine:

2024-01-18 00:34:41 root@laalaa ~
# parted -l | less



Many thanks to all that helped.



YW.  :-)


If you use rsync(1), I suggest using some kind of integrity checking 
tool to verify that the source and destination file systems are 
identical.  I prefer BSD mtree(8):


I assume I'd have to remount the raid like to /raid?
Whew!  That's got more arguments than rsync...



The old /home RAID10 still has its metadata on disk.  I would install 
the "mdadm" package, edit /etc/fstab, copy and rework the old /home line 
(new mount point, add option "ro"), create the mount point, and mount.



David

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

gene heskett wrote:
> > where did the extra 19.4G's come from? Can filesystem
> > ext4's overhead account for that?

In an earlier mail:

> > > command line: rsync -a --bwlimit=10m --fsync --progress /home/ 
> > > /mnt/homevol

David Christensen wrote:
> Please RTFM rsync(1) to choose your options.  These look
> useful:
>--archive, -a   (-rlptgoD)
>--delete
>--hard-links, -H
>--one-file-system, -x
>--sparse, -S

I bet on --hard-links and --sparse as means to avoid the extra disk space
consumption. (--archive is important for other reasons, but it was
already in use as -a with your successful rsync run. --delete will be
of importance if the rsync run gets repeated on the already filled target
directory tree.)

man rsync:

 -H, --hard-links
This tells rsync to look for hard-linked files in the source and
link together the corresponding files on the destination.  With‐
out  this option, hard-linked files in the source are treated as
though they were separate files.
[...]
 -S, --sparse
Try  to  handle  sparse  files  efficiently so they take up less
space on the destination. [...]

One can observe a similar inflation effect when copying the files of a
Debian installation ISO to hard disk. In the original disk directory
on the machine which created the ISO there were hardlinked kernels and
firmware packages. In the ISO these link siblings share the same file
content storage.
But when mounted, the siblings get treated as separate files with
different inode numbers. So the 8,135,584 bytes of the hardlink siblings
  /install.amd/gtk/vmlinuz
  /install.amd/vmlinuz
  /install.amd/xen/vmlinuz
get triplicated when these three files get copied out of the ISO.

I am somewhat astonished that --hard-links is not default in rsync,
as it is quite important for backup fidelity.
(On the other hand it is some effort to find all siblings on the disk.)

Sparse files are files with large areas of 0-bytes. Many filesystems
don't store the zeros but rather an instruction to hand out the given
number of 0-bytes when requested by a reader.


If i were you, i'd let rsync make a complete new copy with --hard-links
--sparse, and --delete, but without --bwlimit= in order to get a higher
copy fidelity and also to check whether the transfer speed really was not
to blame for the appearance of the OOM killer.


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[Charles Curley]

On Tue, 16 Jan 2024 21:10:28 -0500
gene heskett  wrote:

> gene@coyote:~/src/klipper-docs$  lsblk -d -o 
> NAME,MAJ:MIN,MODEL,SERIAL,WWN /dev/sd[hijkl]
> NAME MAJ:MIN MODEL SERIAL WWN
> sdh8:112 Gigastone SSD GSTD02TB230102
> sdi8:128 Gigastone SSD GST02TBG221146
> sdj8:144 Gigastone SSD GST02TBG221146
> sdk8:160 Gigastone SSD GSTG02TB230206
> sdl8:176 Gigastone SSD GSTG02TB230206

Something is seriously wrong here. I worked at Maxtor for a while. They
went out of their way to be sure there were no duplicate serial
numbers.

Gene, I suggest you check these SNs with the SN on the packages (if
there is one) and on the label on the drive.

Also, take each drive, one at a time, attach it to another computer
with a fresh installation of Debian, one you haven't mucked with in any
way, and only one other drive already in it, and read the SNs there.

I also went looking for Gigastone's web site. Every page I tried at
gigastone.com led to what I presume was an Error 404 page. I say
presume because most of the text was in non-English, probably Chinese,
characters.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/18/24 00:50, David Christensen wrote:

On 1/17/24 20:20, gene heskett wrote:

On 1/17/24 19:58, David Christensen wrote:

On 1/17/24 15:58, gene heskett wrote:
Now the question is how did it make this: homevol s/b very close to 
/home  in size but:

root@coyote:~# df && free
Filesystem  1K-blocks  Used  Available Use% Mounted on
udev 16327704 0   16327704   0% /dev
tmpfs 3272684  1912    3270772   1% /run
/dev/sda1   863983352  22348472  797673232   3% /
tmpfs    16363420  1244   16362176   1% /dev/shm
tmpfs    5120 8   5112   1% /run/lock
/dev/sda3    47749868   784   45291076   1% /tmp
/dev/md0p1 1796382580 335102676 1369954928  20% /home
tmpfs 3272684  4956    3267728   1% /run/user/1000
/dev/sdh1  1967892164 354519236 1513336680  19% /mnt/homevol
    total    used    free  shared  
buff/cache available
Mem:    32726840 3417576  515520  934540    30072184 
29309264

Swap:  111902712    2048   111900664
root@coyote:~#

It somehow changed 335G into 354G. Thinking the AppImages dir is 
full of soft links of short names pointing at the long filename and 
had turned the links into duplicates, that was the first thing I 
checked, but it was all good soft-links, so where did the extra 
19.4G's come from? Can filesystem ext4's overhead account for that?



I suggest running rsync(1) with --dry-run, --log-file=FILE, 
--itemize_changes, and whatever other options are needed to find the 
differences.  Please RTFM rsync(1) to choose your options.  These 
look useful:


 --archive, -a    (-rlptgoD)
 --delete

Why --delete?



If you have files on the destination from a previous run of rsync(1) and 
they no longer exist on the source, --delete will get rid of extraneous 
files on the destination.




 --hard-links, -H
 --one-file-system, -x
 --sparse, -S

or --sparse?



First, you need to understand what "sparse file" means:

https://en.wikipedia.org/wiki/Sparse_file


If you have sparse files on the source -- say, 10 GB virtual machine 
images -- then you want rsync(1) to create sparse files on the destination.



Well, my abundance of curiosity, may have killed the cat, but if I 
understand how rsync's -a works, re-running the same command will only 
update for the incoming email and any posts I've made while it was 
running the first time.  So the same command quoted last is now 
running again. when it has exited, which it has now done in about 15 
minutes I'll edit fstab to remove the 60 gigs of swap on md1, remove 
the existing mount of md0p1 as /home taking the raid10 completely out 
of the system. And add the mounting of LABEL=homevolsdh1 as the /home 
partition and reboot. In the event I have to re-install, the raid will 
still contain my data and can be recovered.
I already have a dvd with the most recent netinstall burnt. All I have 
to do is convince it to not install orca and brltty. Probably by 
unplugging _all_ usb stuff except the keyboard and mouse buttons.


What would solve many of my problems is a bit of help from someone who 
it running trinity to tell me how to install it on a system w/o any 
installed gui which obviously disables synaptic. That leaves apt, 
apt-get, and aptitude, unless there is a better way. aptitude is 
uncontrollable, has fixed me once, has torn the system down to another 
install 3 times so the odds are not in my favor.


So those fstab edits have been done, next is a reboot



You should be able to migrate your /home file system from RAID10 to an 
SSD without needing to reinstall Debian.


The migration took two passes because udev can't make up its alleged 
mind so I was finally forced to use the rescue mode to edit fstab to 
mount it by UUID and that worked, I've got /home on the copy right now. 
and I took the 60 G's of swap out too since I've never used more the 20G 
with any gfx program, so I figure 47G's on /dev/sda is enough.  So now 
none of the raid is mounted, but the 30+ second lag when opening a write 
path is still there, so I was erroneously blaming the raid. So I've 
narrowed the problem but w/o a good clue what to do next. One thing that 
bothers me is there is no way the installers parted shows partition 
names for non-raid disks. To me that is a serious bug. It appears from 
the help that it can LABEL a partition but can't read that LABEL. parted 
when asked to print all does that just fine, but the | doesn't put it to 
less, so it scrolls off screen the top 60% of a parted's print all 
output at some fraction of C speed. Not exactly helpful. I have other 
things to do while I cogitate on what to do next.  Many thanks to all 
that helped.



If you use rsync(1), I suggest using some kind of integrity checking 
tool to verify that the source and destination file systems are 
identical.  I prefer BSD mtree(8):


I assume I'd have to remount the raid l

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/17/24 20:20, gene heskett wrote:

On 1/17/24 19:58, David Christensen wrote:

On 1/17/24 15:58, gene heskett wrote:
Now the question is how did it make this: homevol s/b very close to 
/home  in size but:

root@coyote:~# df && free
Filesystem  1K-blocks  Used  Available Use% Mounted on
udev 16327704 0   16327704   0% /dev
tmpfs 3272684  1912    3270772   1% /run
/dev/sda1   863983352  22348472  797673232   3% /
tmpfs    16363420  1244   16362176   1% /dev/shm
tmpfs    5120 8   5112   1% /run/lock
/dev/sda3    47749868   784   45291076   1% /tmp
/dev/md0p1 1796382580 335102676 1369954928  20% /home
tmpfs 3272684  4956    3267728   1% /run/user/1000
/dev/sdh1  1967892164 354519236 1513336680  19% /mnt/homevol
    total    used    free  shared  buff/cache 
available
Mem:    32726840 3417576  515520  934540    30072184 
29309264

Swap:  111902712    2048   111900664
root@coyote:~#

It somehow changed 335G into 354G. Thinking the AppImages dir is full 
of soft links of short names pointing at the long filename and had 
turned the links into duplicates, that was the first thing I checked, 
but it was all good soft-links, so where did the extra 19.4G's come 
from? Can filesystem ext4's overhead account for that?



I suggest running rsync(1) with --dry-run, --log-file=FILE, 
--itemize_changes, and whatever other options are needed to find the 
differences.  Please RTFM rsync(1) to choose your options.  These look 
useful:


 --archive, -a    (-rlptgoD)
 --delete

Why --delete?



If you have files on the destination from a previous run of rsync(1) and 
they no longer exist on the source, --delete will get rid of extraneous 
files on the destination.




 --hard-links, -H
 --one-file-system, -x
 --sparse, -S

or --sparse?



First, you need to understand what "sparse file" means:

https://en.wikipedia.org/wiki/Sparse_file


If you have sparse files on the source -- say, 10 GB virtual machine 
images -- then you want rsync(1) to create sparse files on the destination.



Well, my abundance of curiosity, may have killed the cat, but if I 
understand how rsync's -a works, re-running the same command will only 
update for the incoming email and any posts I've made while it was 
running the first time.  So the same command quoted last is now running 
again. when it has exited, which it has now done in about 15 minutes 
I'll edit fstab to remove the 60 gigs of swap on md1, remove the 
existing mount of md0p1 as /home taking the raid10 completely out of the 
system. And add the mounting of LABEL=homevolsdh1 as the /home partition 
and reboot. In the event I have to re-install, the raid will still 
contain my data and can be recovered.
I already have a dvd with the most recent netinstall burnt. All I have 
to do is convince it to not install orca and brltty. Probably by 
unplugging _all_ usb stuff except the keyboard and mouse buttons.


What would solve many of my problems is a bit of help from someone who 
it running trinity to tell me how to install it on a system w/o any 
installed gui which obviously disables synaptic. That leaves apt, 
apt-get, and aptitude, unless there is a better way. aptitude is 
uncontrollable, has fixed me once, has torn the system down to another 
install 3 times so the odds are not in my favor.


So those fstab edits have been done, next is a reboot



You should be able to migrate your /home file system from RAID10 to an 
SSD without needing to reinstall Debian.



Copying a file system that is mounted read-write is problematic.  It is 
best to remount it read-only, and then copy.  This is hard to do when 
you are logged in and using the file system you want to copy.  Options 
include rebooting into single-user root console or using live media.



To make an exact copy of the source, consider using a tool designed for 
this task -- such as cpio(1), tar(1), or a backup/restore system such as 
amanda(8).



If you use rsync(1), I suggest using some kind of integrity checking 
tool to verify that the source and destination file systems are 
identical.  I prefer BSD mtree(8):


https://manpages.debian.org/bullseye/mtree-netbsd/mtree.8.en.html


(Be careful not to confuse the above with mtree(5) via libarchive.)


David

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/17/24 19:58, David Christensen wrote:

On 1/17/24 15:58, gene heskett wrote:
Now the question is how did it make this: homevol s/b very close to 
/home  in size but:

root@coyote:~# df && free
Filesystem  1K-blocks  Used  Available Use% Mounted on
udev 16327704 0   16327704   0% /dev
tmpfs 3272684  1912    3270772   1% /run
/dev/sda1   863983352  22348472  797673232   3% /
tmpfs    16363420  1244   16362176   1% /dev/shm
tmpfs    5120 8   5112   1% /run/lock
/dev/sda3    47749868   784   45291076   1% /tmp
/dev/md0p1 1796382580 335102676 1369954928  20% /home
tmpfs 3272684  4956    3267728   1% /run/user/1000
/dev/sdh1  1967892164 354519236 1513336680  19% /mnt/homevol
    total    used    free  shared  buff/cache 
available
Mem:    32726840 3417576  515520  934540    30072184 
29309264

Swap:  111902712    2048   111900664
root@coyote:~#

It somehow changed 335G into 354G. Thinking the AppImages dir is full 
of soft links of short names pointing at the long filename and had 
turned the links into duplicates, that was the first thing I checked, 
but it was all good soft-links, so where did the extra 19.4G's come 
from? Can filesystem ext4's overhead account for that?



I suggest running rsync(1) with --dry-run, --log-file=FILE, 
--itemize_changes, and whatever other options are needed to find the 
differences.  Please RTFM rsync(1) to choose your options.  These look 
useful:


 --archive, -a    (-rlptgoD)
 --delete

Why --delete?

 --hard-links, -H
 --one-file-system, -x
 --sparse, -S

or --sparse?

Well, my abundance of curiosity, may have killed the cat, but if I 
understand how rsync's -a works, re-running the same command will only 
update for the incoming email and any posts I've made while it was 
running the first time.  So the same command quoted last is now running 
again. when it has exited, which it has now done in about 15 minutes 
I'll edit fstab to remove the 60 gigs of swap on md1, remove the 
existing mount of md0p1 as /home taking the raid10 completely out of the 
system. And add the mounting of LABEL=homevolsdh1 as the /home partition 
and reboot. In the event I have to re-install, the raid will still 
contain my data and can be recovered.
I already have a dvd with the most recent netinstall burnt. All I have 
to do is convince it to not install orca and brltty. Probably by 
unplugging _all_ usb stuff except the keyboard and mouse buttons.


What would solve many of my problems is a bit of help from someone who 
it running trinity to tell me how to install it on a system w/o any 
installed gui which obviously disables synaptic. That leaves apt, 
apt-get, and aptitude, unless there is a better way. aptitude is 
uncontrollable, has fixed me once, has torn the system down to another 
install 3 times so the odds are not in my favor.


So those fstab edits have been done, next is a reboot



David

.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[David Wright]

On Wed 17 Jan 2024 at 15:34:09 (-0500), gene heskett wrote:
> On 1/17/24 12:27, Thomas Schmitt wrote:
> > David Christensen wrote:
> > > I suspect the conflicting serial numbers are causing problems in the 
> > > kernel,
> > > as indicated by the /dev/disk/by-id/* problems.
> > 
> > That's not in the kernel but in udev/systemd's process of creating the
> > symbolic links in /dev/disk/by-id/.
> > It gets /dev/sd[h-l] and /dev/sd[h-l]1 as kernel generated device files.
> > But sd[ij] and also sd[hl] show pair-wise the same serial numbers.
> > In case of sd[ij] the outcome is mixed: links to sdi and sdj1 survive.
> > In case of sd[hl] we see a less strange outcome: sdh and sdh1, while
> > sdl and sdl1 are missing.
> > 
> missing because the original command line did not look at sdl.
> I added the l and it showed up. No magic.

What do you mean, it was "missing"? The original command, which I wrote
for you, contained a wildcard, so it doesn't miss anything that's there:

  root@coyote:~# for j in /dev/disk/by-id/* ; do printf '%s\t%s\n' "$(realpath 
"$j")" "$j" ; done 

and there was no sdl in the output from that command. In fact,
there was no "l" in your post between the "l" in "realpath",
above, and the "l" in "like", below:

  root@coyote:~#
  but like I wrote, 2 pairs with identical "serial numbers", so the

https://lists.debian.org/debian-user/2024/01/msg00658.html
shows this, that no sdl was seen under by-id/.

> > The open question (at least to me) is whether it's the disks or the
> > controllers or the drivers which cause the duplication.
> Neither, a typu in the original command.

Cheers,
David.

Re: To partition or not to partition MD arrays (Was Re: smartctl cannotaccess my storage, need syntax help)

[gene heskett]

On 1/17/24 19:54, Steve McIntyre wrote:

Andy Smith wrote:


The newer set of people recommending partitions are mostly doing so
because there's been a few incidents of "helpful" PC motherboards
detecting on boot what they think is a corrupt GPT, and replacing it
with a blank one, damaging the RAID. This is a real thing that has
happened to more than one person; it even got linked on Hacker News
I believe.

Then there will just be people going by taste.

Personally I still put them directly on drives. If I ever get taken
out by one of those crappy motherboards, I reserve the right to get
a different religion. 


I'm clearly a member of a third group of people,,, :-)

Putting partitions on the RAID drives helps *me* identify them.


you aren't alone Steve.
Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/17/24 15:58, gene heskett wrote:
Now the question is how did it make 
this: homevol s/b very close to /home  in size but:

root@coyote:~# df && free
Filesystem  1K-blocks  Used  Available Use% Mounted on
udev 16327704 0   16327704   0% /dev
tmpfs 3272684  1912    3270772   1% /run
/dev/sda1   863983352  22348472  797673232   3% /
tmpfs    16363420  1244   16362176   1% /dev/shm
tmpfs    5120 8   5112   1% /run/lock
/dev/sda3    47749868   784   45291076   1% /tmp
/dev/md0p1 1796382580 335102676 1369954928  20% /home
tmpfs 3272684  4956    3267728   1% /run/user/1000
/dev/sdh1  1967892164 354519236 1513336680  19% /mnt/homevol
    total    used    free  shared  buff/cache 
available
Mem:    32726840 3417576  515520  934540    30072184 
29309264

Swap:  111902712    2048   111900664
root@coyote:~#

It somehow changed 335G into 354G. Thinking the AppImages dir is full of 
soft links of short names pointing at the long filename and had turned 
the links into duplicates, that was the first thing I checked, but it 
was all good soft-links, so where did the extra 19.4G's come from? Can 
filesystem ext4's overhead account for that?



I suggest running rsync(1) with --dry-run, --log-file=FILE, 
--itemize_changes, and whatever other options are needed to find the 
differences.  Please RTFM rsync(1) to choose your options.  These look 
useful:


--archive, -a   (-rlptgoD)
--delete
--hard-links, -H
--one-file-system, -x
--sparse, -S


David

Re: To partition or not to partition MD arrays (Was Re: smartctl cannot access my storage, need syntax help)

[Steve McIntyre]

Andy Smith wrote:
>
>The newer set of people recommending partitions are mostly doing so
>because there's been a few incidents of "helpful" PC motherboards
>detecting on boot what they think is a corrupt GPT, and replacing it
>with a blank one, damaging the RAID. This is a real thing that has
>happened to more than one person; it even got linked on Hacker News
>I believe.
>
>Then there will just be people going by taste.
>
>Personally I still put them directly on drives. If I ever get taken
>out by one of those crappy motherboards, I reserve the right to get
>a different religion. 😀

I'm clearly a member of a third group of people,,, :-)

Putting partitions on the RAID drives helps *me* identify them.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
Can't keep my eyes from the circling sky,
Tongue-tied & twisted, Just an earth-bound misfit, I...

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/17/24 16:45, David Christensen wrote:

On 1/17/24 12:30, gene heskett wrote:

By LABELing the partitions uniquely, that problem so far as I can
see, is solved.



Okay.


So, are you confident that your motherboard ports, HBA ports, and SSD's
are all working correctly now?



The OOM death of the system was the xfce4 terminal apparently being
set for unlimited scrollback and that was eating the memory. Switching 
to Konsole with has the ability to control the scrollback

to 200 lines, and its taken all 32G's as .cache and 1536 1k blocks of
swap, and its working w/o any OOM actions I've detected.



Okay.


Xfce -> Terminal Emulator -> right click on screen -> Preferences ->
General -> Scrolling:

 Scrollback    200
 Unlimited scrollback    uncheck


Using tee(1) would allow you to both monitor progress and save standard 
output and/or standard error (via shell redirection).



A related issue is that lots of standard output can slow a program. 
Minimizing a terminal can help.  Redirecting standard output to a file 
or to /dev/null can help, especially when done on the remote host while 
using ssh(1).



The best solution is to tell rsync(1) not to generate messages on 
standard output -- do not use --verbose, do not use --info, do not use 
--progress, etc.; use --quiet, etc..


All good hints after it is done.  Now the question is how did it make 
this: homevol s/b very close to /home  in size but:

root@coyote:~# df && free
Filesystem  1K-blocks  Used  Available Use% Mounted on
udev 16327704 0   16327704   0% /dev
tmpfs 3272684  19123270772   1% /run
/dev/sda1   863983352  22348472  797673232   3% /
tmpfs16363420  1244   16362176   1% /dev/shm
tmpfs5120 8   5112   1% /run/lock
/dev/sda347749868   784   45291076   1% /tmp
/dev/md0p1 1796382580 335102676 1369954928  20% /home
tmpfs 3272684  49563267728   1% /run/user/1000
/dev/sdh1  1967892164 354519236 1513336680  19% /mnt/homevol
   totalusedfree  shared  buff/cache 
available
Mem:32726840 3417576  515520  93454030072184 
29309264

Swap:  1119027122048   111900664
root@coyote:~#

It somehow changed 335G into 354G. Thinking the AppImages dir is full of 
soft links of short names pointing at the long filename and had turned 
the links into duplicates, that was the first thing I checked, but it 
was all good soft-links, so where did the extra 19.4G's come from? Can 
filesystem ext4's overhead account for that?


David


Thanks David.

.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/17/24 12:30, gene heskett wrote:

By LABELing the partitions uniquely, that problem so far as I can
see, is solved.



Okay.


So, are you confident that your motherboard ports, HBA ports, and SSD's
are all working correctly now?



The OOM death of the system was the xfce4 terminal apparently being
set for unlimited scrollback and that was eating the memory. 
Switching to Konsole with has the ability to control the scrollback

to 200 lines, and its taken all 32G's as .cache and 1536 1k blocks of
swap, and its working w/o any OOM actions I've detected.



Okay.


Xfce -> Terminal Emulator -> right click on screen -> Preferences ->
General -> Scrolling:

Scrollback  200
Unlimited scrollbackuncheck


Using tee(1) would allow you to both monitor progress and save standard 
output and/or standard error (via shell redirection).



A related issue is that lots of standard output can slow a program. 
Minimizing a terminal can help.  Redirecting standard output to a file 
or to /dev/null can help, especially when done on the remote host while 
using ssh(1).



The best solution is to tell rsync(1) not to generate messages on 
standard output -- do not use --verbose, do not use --info, do not use 
--progress, etc.; use --quiet, etc..



David

Re: smartctl cannot access my storage, need syntax help

[gene heskett]

On 1/17/24 16:16, Thomas Schmitt wrote:

Hi,

i wrote:

What did finally help ? Just the shorter terminal scroll back memory ?


gene heskett wrote:

That, and possibly the --bwlimit=10m, giving the SSD time to keep their
stuff in one sock.


Then i place my bet on the terminal alone.
Linux is able to handle disk-to-disk copies that are larger than the
available memory. This is a standard use case.



How large was it set when your runs caused the OOM killer to act ?



different terminal, xfce4's is apparently unlimited but can't find it in the
config prefs.


I normally start new xterms by

   xterm -ls -geometry 80x24 -bg wheat -fg black -sl 1 +sb &

The -sl option gives the number of lines to be memorized for scrollback.
Black-on-wheat is a calmative color combination which does not overwork
the eyes.


Thank you, I did not know that.


Have a nice day :)

Thomas

.


Cheers, Gene Heskett.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

i wrote:
> > What did finally help ? Just the shorter terminal scroll back memory ?

gene heskett wrote:
> That, and possibly the --bwlimit=10m, giving the SSD time to keep their
> stuff in one sock.

Then i place my bet on the terminal alone.
Linux is able to handle disk-to-disk copies that are larger than the
available memory. This is a standard use case.


> > How large was it set when your runs caused the OOM killer to act ?

> different terminal, xfce4's is apparently unlimited but can't find it in the
> config prefs.

I normally start new xterms by

  xterm -ls -geometry 80x24 -bg wheat -fg black -sl 1 +sb &

The -sl option gives the number of lines to be memorized for scrollback.
Black-on-wheat is a calmative color combination which does not overwork
the eyes.


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[David Christensen]

On 1/17/24 09:31, Thomas Schmitt wrote:

Hi,

David Christensen wrote:

I suspect the conflicting serial numbers are causing problems in the kernel,
as indicated by the /dev/disk/by-id/* problems.


That's not in the kernel but in udev/systemd's process of creating the
symbolic links in /dev/disk/by-id/.
It gets /dev/sd[h-l] and /dev/sd[h-l]1 as kernel generated device files.
But sd[ij] and also sd[hl] show pair-wise the same serial numbers.
In case of sd[ij] the outcome is mixed: links to sdi and sdj1 survive.
In case of sd[hl] we see a less strange outcome: sdh and sdh1, while
sdl and sdl1 are missing.

The open question (at least to me) is whether it's the disks or the
controllers or the drivers which cause the duplication.



Thank you for the explanation.


I would still remove them.


David