Thank you for this Kay, And thanks to others for helpful suggestions on
and off-list as well.
As for Kay's plan, as a matter of fact, I spent about a week doing this
already. Managed to get one of the three critical drives completely
imaged to a file on my scratch array, but no software reset tools worked
for this failture mode. The drives need to actually have no power to
reset their "firmware clock" or whatever it is. So I had to keep
physically plugging and unplugging the power cable by hand. Eventually I
started to worry about how many times they drives would surivive a cold
start. They are rated for a lot of power cycles, but it is a common
point of danger.
I can set the computer to keep powering itself off an on again, but if
I'm doing that I might as well have the array up and ready to copy my
most important files first. Nearly everything is on tape, but the most
recent is the most interesting. Something that bit me after trying a
new raid card is the raid card itself remembered that the drives had
failed, and makred them as "bad". even old card refused to re-assemble
the array. That was another tough week. But, after experimenting with a
stack of old 3 TB disks I learned how to reproduce the exact broken
config I had and how to get out of it. It is possible to store and
replace the raid config areas from the drives, but in the end, a
well-planned "recreate", which overwrites the RAID config area on all
the drives, is what I'm doing now every time I power up that box.
In these efforts I discovered a few things that could be generally
useful. One of the reasons I wasn't doing backups as often as I should
is because the process of discovering newly-created files is so costly.
Even if there aren't that many files to move, an rsync on a 60-megafiile
file system can take 18 hours, and slows things down that whole time.
Better ways are:
1) Turns out xfs has a feature called a "bulkstat" that lets you
retrieve all the date stamps in the whole file system in a few minutes,
tied to inodes, which you can then connect the newest ones to actual
path names. 10 min instead of hours.
2) On zfs, the thing is to use snapshots and then ask "zfs diff" to tell
you what has changed since a snapshot. Again minutes instead of hours.
I was surprised to find that there are no off-the-shelf tools for
taking advantage of thsee features.
I plan to do these in the future so that I have a near-real-time copy of
all most recently-created files to my tape backup cache disk.
An before you ask, yes this has all been horrible. Hope this info ends
up useful to someone!
And again, thanks for all the help, CCP4BB!
-James Holton
MAD Scientist
On 8/27/2026 11:38 PM, Kay Diederichs wrote:
Hi James,
the following does not need new firmware (it is thus not an answer to your
question), and could be done for each of your 8TB drives:
1) put only one of the old drives (to be called /dev/sdb below), and a new
(8TB) drive (e.g. /dev/sdc), into some machine
2) figure out how much you can copy in <28 minutes with dd. Maybe bs=1M
count=300000 would work , assuming the drive can read/write 200MB/sec - if not
modify bs (which should ideally be a divider of the disk size). bs=1M
count=300000 would be 300GB .
3) copy sdb fully but piecewise to sdc:
- copy 300GB with a single dd command like dd if=/dev/sdb of=/dev/sdc
- wait a few seconds until the new drive (sdc) has emptied its cache; better:
sync
- cycle power of the old drive (better to have it on its own SATA controller,
but that might not matter)
- copy the exact next 300GB with dd, using dd if=/dev/sdb of=/dev/sdc bs=1M
count=300000 skip=300000 seek=300000
- repeat the "sync ; powercycle; dd" cycle until fully copied
Maybe the "powercycle" can be replace by "hdparm -w" or some other command ("udisks
--detach /dev/sdb" ?). In that case, the whole procedure could be scripted.
Hope this helps,
Kay
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
