> On Jul 31, 2026, at 12:40 AM, Tom Dial <[email protected]> wrote: > > > On 7/30/26 14:49, Casey Deccio wrote: >>> On Jul 30, 2026, at 1:04 PM, Alexander V. Makartsev <[email protected]> >>> wrote: >>> >>> On 7/30/26 18:17, Casey Deccio wrote: >>>>> On Jul 29, 2026, at 5:28 PM, Andy Smith<[email protected]> wrote: >>>>> >>>>> ... >>>>> If it does occur when /home is mounted, then it is the disks in the RAID >>>>> setup or the RAID setup itself. Does each individual disk perform >>>>> anomalously? >>>>> >>>>> "fio" is a good tool,to carry out performance testing of storage >>>>> devices. Have a look what the random 4k read and write I/O is like in >>>>> terms of IOPS, and the sequential disk transfer rate in MB/s for reads >>>>> or writes. You can try that on the array and then break the array and >>>>> try it on each individual disk. Obviously this is destructive to the >>>>> data. >>>>> >>>>> If the performance is as expected on a per-disk basis but terrible only >>>>> with RAID, it may be time to post to the linux-raid mailing list as this >>>>> is unlikely to be a Debian-specific issue. I would then also be checking >>>>> if it happens in other RAID configurations like RAID-1. >>>>> >>>>> 150 IOPS of random reads (not coming from RAM) is reasonable for a >>>>> single 7.2kRPM SATA disk. You could expect 2-3x this from a RAID-5 of >>>>> three identical disks. >>>>> >>>>> Write IOPS is a bit more difficult to guess and will depend on whether >>>>> you have disabled the write cache in the drives themselves. Due to the >>>>> RAID-5 write penalty (read-modify-write) you may expect only 50-70% of a >>>>> single disk's IOPS for random writes. >>>> Thanks for that very helpful information. I've run fio on the >>>> still-assembled array, and here's what I get for read/write: >>>> >>>> read: IOPS=237 >>>> write: IOPS=79 >>>> >>>> I'm not sure exactly what to make of that... >>> I might add insult to the injury, but are you sure you need RAID-5 if it >>> consists of just 3 drives? >>> Usually around 30% per drive's capacity will be sacrificed to hold error >>> correction data for the array, >>> so maybe it will be much better to use 2 drives in RAID-1 (Mirror) >>> configuration, if you need decent write performance. >>> Third drive could be used as spare device for the array, or a separate >>> storage without redundancy. >> Great ideas. I originally did RAID5 because I wasn't sure how many drives I >> wanted in the array, and I wondered if I might grow the array at some point. >> But I'll consider RAID 1 if I settle on three drives. > > A bit of analysis suggests this is a low performance server, mainly if not > entirely because of the disks. The server board surely is not the bottleneck. > As an earlier post suggested, the sync times noted are quite as should be > expected. > > The bottleneck almost certainly is the 6GBit/Sec consumer disks. The > 6GBits/Sec transfer speed is only attainable on the cables from the system > board to the individual disks. It implies a maximum transfer rate around 600 > MBytes/Sec. but in sustained transfers, the disks cannot write that fast. AI > query suggests 100-180 MB/Sec (depending on what part of the disk is being > accessed) for CMR disks and 10-30 MB/Sec(!) for SMR disks. Raid 5 with 3 > disks will reduce those to around 50% -75%, around 8-24 MB/sec so the > expected average sustained write speed is unlikely to be more than 100MB/Sec. > even with CMR disks. That is about, or slightly above, what is available from > a Gigabit Ethernet transfer; for SMR disks, expect no more than around 25 > MB/Sec. Large writes will take a while. > > The pattern of sync times reported is consistent with that, suggesting that > rsync makes the data available for writing somewhat faster than it can be > written to the disk surfaces. Note that this doesn't apply to reading; the > RAID will read and deliver data roughly twice as fast as a single disk and > continue to do so on sustained reads as long as the data read is consumed at > least as fast. Once the bulk data is transferred, performance may be > satisfactory even with the low-spec drives. > > The system with the Megaraid card would appear to perform better due to its > internal cache: it will report write completion as soon as the data is in its > cache (thus unblocking sync) and handle transfer to the disks internally. AI > suggestion for the new server: set Write-Back rather than Write-Through for > PCH Cache settings if that is not already done; that will allow the Intel > driver to treat main memory as cache. It will not, however, make the data > transfer any more quickly to the disks in the array. > > AI was used here, but not extensively.
Thanks for the assessment. And also thanks to all the other respondents for their contributions. I can't really change the hardware I have, but this is helpful for figuring out how to move forward, both with my current system and for future system design. Casey
