On 7/30/26 14:49, Casey Deccio wrote:
On Jul 30, 2026, at 1:04 PM, Alexander V. Makartsev <[email protected]> wrote:
On 7/30/26 18:17, Casey Deccio wrote:
On Jul 29, 2026, at 5:28 PM, Andy Smith<[email protected]> wrote:
...
If it does occur when /home is mounted, then it is the disks in the RAID
setup or the RAID setup itself. Does each individual disk perform
anomalously?
"fio" is a good tool,to carry out performance testing of storage
devices. Have a look what the random 4k read and write I/O is like in
terms of IOPS, and the sequential disk transfer rate in MB/s for reads
or writes. You can try that on the array and then break the array and
try it on each individual disk. Obviously this is destructive to the
data.
If the performance is as expected on a per-disk basis but terrible only
with RAID, it may be time to post to the linux-raid mailing list as this
is unlikely to be a Debian-specific issue. I would then also be checking
if it happens in other RAID configurations like RAID-1.
150 IOPS of random reads (not coming from RAM) is reasonable for a
single 7.2kRPM SATA disk. You could expect 2-3x this from a RAID-5 of
three identical disks.
Write IOPS is a bit more difficult to guess and will depend on whether
you have disabled the write cache in the drives themselves. Due to the
RAID-5 write penalty (read-modify-write) you may expect only 50-70% of a
single disk's IOPS for random writes.
Thanks for that very helpful information. I've run fio on the still-assembled
array, and here's what I get for read/write:
read: IOPS=237
write: IOPS=79
I'm not sure exactly what to make of that...
I might add insult to the injury, but are you sure you need RAID-5 if it
consists of just 3 drives?
Usually around 30% per drive's capacity will be sacrificed to hold error
correction data for the array,
so maybe it will be much better to use 2 drives in RAID-1 (Mirror)
configuration, if you need decent write performance.
Third drive could be used as spare device for the array, or a separate storage
without redundancy.
Great ideas. I originally did RAID5 because I wasn't sure how many drives I
wanted in the array, and I wondered if I might grow the array at some point.
But I'll consider RAID 1 if I settle on three drives.
A bit of analysis suggests this is a low performance server, mainly if not
entirely because of the disks. The server board surely is not the bottleneck.
As an earlier post suggested, the sync times noted are quite as should be
expected.
The bottleneck almost certainly is the 6GBit/Sec consumer disks. The 6GBits/Sec
transfer speed is only attainable on the cables from the system board to the
individual disks. It implies a maximum transfer rate around 600 MBytes/Sec. but
in sustained transfers, the disks cannot write that fast. AI query suggests
100-180 MB/Sec (depending on what part of the disk is being accessed) for CMR
disks and 10-30 MB/Sec(!) for SMR disks. Raid 5 with 3 disks will reduce those
to around 50% -75%, around 8-24 MB/sec so the expected average sustained write
speed is unlikely to be more than 100MB/Sec. even with CMR disks. That is
about, or slightly above, what is available from a Gigabit Ethernet transfer;
for SMR disks, expect no more than around 25 MB/Sec. Large writes will take a
while.
The pattern of sync times reported is consistent with that, suggesting that
rsync makes the data available for writing somewhat faster than it can be
written to the disk surfaces. Note that this doesn't apply to reading; the RAID
will read and deliver data roughly twice as fast as a single disk and continue
to do so on sustained reads as long as the data read is consumed at least as
fast. Once the bulk data is transferred, performance may be satisfactory even
with the low-spec drives.
The system with the Megaraid card would appear to perform better due to its
internal cache: it will report write completion as soon as the data is in its
cache (thus unblocking sync) and handle transfer to the disks internally. AI
suggestion for the new server: set Write-Back rather than Write-Through for PCH
Cache settings if that is not already done; that will allow the Intel driver to
treat main memory as cache. It will not, however, make the data transfer any
more quickly to the disks in the array.
AI was used here, but not extensively.
Regards,
Tom Dial
Thanks,
Casey
