On 17/02/2024 03:14, Robert Nichols wrote:
On 2/16/24 08:44, Dominic Raferd wrote:
Until then, I am interested in your parallel processing approach.
Presumably you start 8 parallel rdiff-backup verify sessions for
datetime points -1 to -8 (and then, when they are all complete, -9 to
-16, -17 to -23...)? And you run 8 in parallel because your CPU has 8
cores?
I have 16 cores, actually, but by experiment I found that 8 parallel
threads seems to be the sweet spot. I don't know how much of that is
unique to my system and the nature of my backups. I did have to add a
pre-scan of the file_statistics metadata files to look for increment
sizes of 1GB or greater, and limit the number of parallel checks to 1
if any are found. All it takes is one huge ISO file in the increments
to gobble up cache and make the parallel checks really slow. I haven't
spent much time trying to tune that adjustment, and all the
experimenting was done back when I had just 32GB of RAM.
I let the parallel threads run independently, without waiting for
anything in the others. Effectively, I run the threads with the level
sequences:
{-1..-99..8}
{-2..-99..8}
{-3..-99..8}
...
{-8..-99..8}
and then just wait for everything to complete.
The code is really nothing like that, but that is the effect. You
might expect the threads to get badly out of sync, but because of the
effects of I/O caching, whichever threads finish a step first find
themselves slowed down by I/O waits more than do the threads that
advance to a new step later. The threads tend to stay quite
beautifully in sync. Again, that's on my system with my backups. YMMV.
Very interesting. A while ago I set my timedicer-verify script to run
verifications in parallel but it seemed to make everything slower not
faster, admittedly when running on a much less powerful (and virtual)
machine than yours, so I stripped out all that code. But I should look
at it again (I guess I must have backups!)...
