On 2/15/24 07:41, The Wanderer wrote:
On 2024-02-15 at 03:09, David Christensen wrote:
On 2/14/24 18:54, The Wanderer wrote:
On 2024-01-09 at 14:22, The Wanderer wrote:
On 2024-01-09 at 14:01, Michael Kjörling wrote:
On 9 Jan 2024 13:25 -0500, from The Wanderer
I've ordered a 22TB external drive
Make? Model? How it is interfaced to your computer?
It's a WD Elements 20TB drive (I'm not sure where I got the 22 from);
the back of the case has the part number WDBWLG0200HBK-X8 (or possibly
-XB, the font is kind of ambiguous). The connection, per the packaging
label, is USB-3.
Okay.
STFW it seems that drive uses CMR, which is good:
https://nascompares.com/answer/list-of-wd-cmr-and-smr-hard-drives-hdd/
The big change of plans in the middle of my month-plus process was the
decision to replace the entire 8-drive array with a 6-drive array, and
the reason for that was because the 8-drive array left me with no open
SATA ports to be able to connect spare drives in order to do drive
replacements without needing to rebuild the whole shaboozle.
Having spare drive bays for RAID drive replacement is smart.
If you have a processor, memory, PCIe slot, and HBA to match those
SSD's, the performance of those SSD's should be very nice.
The CPU is a Ryxen 5 5600X. The RAM is G-Skill DDR4 2666MHz, in two 32GB
DIMMs. I don't know how to assess PCIe slots and HBA, but the
motherboard is an Asus ROG Crosshair VIII Dark Hero, which I think was
the top-of-the-line enthusiast motherboard (with the port set my
criteria called for) the year I built this machine.
I'm pretty sure my performance bottleneck for most things is the CPU (or
the GPU, where that comes into play, which here it doesn't);
storage-wise this seems so far to be at least as fast as what I had
before, but it's hard to tell if it's faster.
It would not surprise me if the Intel D3-S4510 server drives are
somewhat slower than the Samsung EVO 870 desktop drives. But the Intel
disks are designed to pull a heavy load all day for years on end.
Do you have a tool to monitor disk throughput and utilization? I use
Xfce panel Disk Performance Monitor applets and nmon(1) in a Terminal.
Those plus CPU and memory monitoring tools should allow you to determine
if your workload is CPU bound, memory bound, or I/O bound.
The key concept is "data lifetime". (Or alternatively, "destruction
policy".)
I can see that for when you have a tiered backup structure, and are
looking at the lifetimes of each backup copy. For my live system, my
intended data lifetime (outside of caches and data kept in /tmp) is
basically "forever".
I try to group my data in anticipation of backup, etc., requirements.
When I get it right, disaster preparedness and disaster recovery are easier.
I believe ZFS can do more hard links. (Much more? Limited by
available storage space?)
I'm not sure, but I'll have to look into that, when I get to the point
of trying to set up that tiered backup.
...
... without [rsnapshot hard link]
deduplication there wouldn't have been enough space on the drive
for more than the single copy, ...
ZFS provides similarly useful results with built-in compression and
de-duplication.
I have the impression that there are risk and/or complexity aspects to
it ...
Of course. ZFS is sophisticated storage technology. It looks
deceptively simple when you are window shopping, but becomes non-trivial
once you put real data on it, have to live with it 24x7, have to prepare
for disasters, and have to recover from disasters. There is a lot to
learn and "more than enough rope to shoot yourself in the foot".
That sounds like an N-way merge problem ...
It does sound like that, yes. I'm already aware of jdupes, and of a few
other tools (part of the work I already did in getting this far was
rdfind, which is what I used to set up much of the hardlink
deduplication that wound up biting me in the butt), but have not
investigated LVM snapshot - and the idea of trying to script something
like this, without an existing known-safe copy of the data to fall back
on, leaves me *very* nervous.
Figuring out how to be prepared to roll back is the other uncertain and
nervous-making part. In some cases it's straightforward enough, but
doing it at the scale of the size of those copies is at best daunting.
https://html.duckduckgo.com/html?q=lvm%20snapshot%20restore
Use another computer or a VM to learn and practice LVM snapshots and
restores, then use those skills when doing the N-way merge.
out of the way, shutting down all parts of the system that might be
writing to the affected filesystems, and manually copying out the
final state of the *other* parts of those filesystems via rsync,
bypassing rsnapshot. That was on Saturday the 10th.
Then I grabbed copies of various metadata about the filesystems,
the LVM, and the mdraid config; modified /etc/fstab to not mount
them; deactivated the mdraid, and commented it out of
/etc/mdadm/mdadm.conf; updated the initramfs; shut down; pulled all
eight Samsung 870 EVO drives; installed six brand-new Intel
data-center-class (or so I gather) SSDs;
Which model? What size?
lshw says they're INTEL SSDSCK2B03. The packaging says SSDSCK2B038T801.
Nice.
IIRC, the product listing said they were 3.84 TB (or possibly TiB). lshw
says 'size: 3567GiB (3840GB)'. IIRC, the tools I used to partition them
and build the mdraid and so forth said 3.84 TB/TiB (not sure which), or
3840 GB/GiB (same).
For comparison, the 870 EVO drives - which were supposed to be 2TB
apiece - were reported by some of those same tools as exactly 2000 of
the same unit.
This does mean that I have more total space available in the new array
than in the old one,
8 @ 2 TB disks in RAID6 should provide 12 TB of capacity.
6 @ 3.84 TB disks in RAID6 should provide 15.36 TB of capacity.
but I've tried to allocate only as much space as
was in the old array, insofar as I could figure out how to do that in
the limited environment I was working in. (The old array and/or LV setup
had sizes listed along the lines of '<10TiB', but my best attempt at
replicating it gave something which reports sizes along the lines of
'10TiB', so I suspect that my current setup is actually slightly too
large to fit on the old disks.)
LVM should give you the ability to resize logical volumes as required
going forward.
Data integrity validation is tough without a mechanism. Adding an
rsnapshot(1) postexec MD5SUMS, etc., file into the root of each
backup tree could solve this need, but could waste a lot of time and
energy checksumming files that have not changed.
AFAIK, all such things require you to be starting from a point with a
known-good copy of the data, which is a luxury I don't currently have
(as far as validating my current data goes). It's something to keep in
mind when planning a more proper backup system, however.
One of the reasons I switched to ZFS was because ZFS has built-in
data and metadata integrity checking (and repair; depending upon
redundancy).
I'm not sure I understand how this would be useful in the case I have at
hand; that probably means that I'm not understanding the picture properly.
Bit rot is the enemy of forever:
https://html.duckduckgo.com/html?q=bit%20rot
The sooner you have MD5SUMS, etc., the sooner you can start monitoring
for file damage by any means. The rsnapshot(1) community may already
have a working solution.
(This does leave me without having restored the read-only backup
data from the old system state. I care less about that; I'll want
it eventually, but it isn't important enough to warrant postponing
getting the system back in working order.)
I do still want/need to figure out what to do about an *actual*
backup system, to external storage, since the rsnapshot thing
apparently isn't going to be viable for my circumstance and use
case. There is, however, now *time* to work on doing that, without
living under the shadow of a known immediate/imminent data-loss
hardware failure.
rsync(1) should be able to copy backups onto an external HDD.
Yeah, but that only provides one tier of backup;
It appears I misunderstood.
So, live data on 6 @ Intel SSD's and rsnapshot(1) backups on the 20 GB
WD Elements USB HDD?
the advantage of
rsnapshot (or similar) is the multiple deduplicated tiers, which gives
you options if it turns out the latest backup already included the
damage you're trying to recover from.
If rsnapshot(1) is your chosen backup tool, you will want to learn
everything you can about it. Beyond RTFM rsnapshot(1), STFW I see:
https://rsnapshot.org/rsnapshot/docs/docbook/rest.html
http://www2.rsnapshot.org/
https://sourceforge.net/p/rsnapshot/mailman/rsnapshot-discuss/
(USB-3 will almost certainly not be a viable option for an automatic
scheduled backup of the sort rsnapshot's documentation suggests, because
the *fastest* backup cycle I saw from my working with the data I had was
over three hours, and the initial pass to copy the data out to the drive
in the first place took nearly *20* hours. A cron job to run even an
incremental backup even once a day, much less the several times a day
suggested for the deeper rsnapshot tiers, would not be *remotely*
workable in that sort of environment. Though on the flip side, that's
not just a USB-3 bottleneck, but also the bottleneck of the spinning
mechanical hard drive inside the external case...)
I think the Raspberry Pi, etc., users on this list live with USB storage
and have found it to be reliable enough for personal and SOHO network use.
I have used the rsync(1) command for many years, both interactively and
via scripts, but RTFM indicates rsync(1) can be run as a server. I
wonder if that would help performance, as a service can cache things
that a command must find on every run (?).
David
