Hello, rdiff-backup has many advantages but speed isn't one of those. I'm sure we can improve the situation but this takes time. Kr, Eric
On 30 May 2025 16:20:29 CEST, Duane Abrames <du...@spaztic.net> wrote: >(I sent this yesterday, but have not seen it come back to me, nor in the >online archives. Not sure if there is some additional approval process, or >if I was just too quick to send this after joining. Apologies if it makes >a duplicate.) > >Hello, new to the list and rdiff-backup. > >First let me describe my situation. I have just started using rdiff-backup >to back up my home server, which contains various music, videos, ebooks, >comics, and audio books ( 34.0 TB in 29,244 folders and 460,277 files). >The source and target servers are both Unraid boxes, so I am running >rdiff-backup in a Docker container. I had initially set up just one >container on the target server, and just mounted the source data via NFS. >My initial backup finished last night, coming in at about 6.5 days. There >seemed to be a lot of latency involved in copying the files; I was watching >files get created, and I watched a directory with 40 or so files of less >than 10mb each take 5 minutes or more to copy, and this was the 'initial' >backup, so there would be no comparison, just copying. >I suspected that performance would be better over SSH than NFS, so I added >a container to the source server and reconfigured things so that the paths >over ssh would match the former NFS paths. After setting all that up, I >kicked off an increment. That increment has been running for 10 hours now, >and using rdiffweb I can see that it is going through checking the folders, >because the version times change from 10 hours ago (before checking) to 7 >days ago (after). It looks like even an increment will take days. I was >hoping to get at least one a day, but if this is the best I can do, I might >have to fall back to weekly backups, or evaluate another option. I wish I >had started this increment with additional verbosity so I might be able to >tell what it's actually doing. I feel like at this speed, it must be doing >checksums or full comparisons, since I can get the 'metadata' (file time, >size, owner, etc) for the whole share in about 3 minutes with ls -lR. Does >my experience seem typical, and if not, how should I begin nailing down the >source of the issue? I kind of assumed that since there were probably less >than 200 new/changed files, the increment would be measured in minutes (or >at worst hours) and not days. > >Some system specs, just so everyone knows I'm not trying to run this on a >Dorito chip: >Source: AMD 5600G / 32gb RAM / 7.2k spinning SATA disk x 9 (with parity) >on IBM SAS controller >Target: AMD 5500 / 64gb RAM / 7.2k spinning SATA disk x6 (without parity) >on mobo / add-in SATA. >Connectivity between the two is gigabit ethernet using a dumb switch. >Servers are less than 10 feet apart, so cable length might be 20 or 30 feet >since cables run under the house. > >I don't have a lot of tools inside the containers to troubleshoot with, but >they are being pulled from my personal github container repo, so I can >always add things and rebuild as needed. The base OS in the container is >ubuntu/latest. I went into this project excited (as excited as one can be >about backups anyway,) as having increments to go back to, and still having >the filesystem be usable as-is seemed like eating the cake and keeping it >too (Is the cake a lie?) I am less enthused now, and wondering if I will >end up with a plain rsync (which is basically what I had a few months ago >before some hardware deaths) or using a backup product that packages >backups in its own native format so that any use or restore from the system >requires the use of the backup software. > >Thanks for your time and attention. >-Duane
