Hello Duffer, My first guess is it's running regression. It's a long process and almost all the files are reverified. Could you confirm which version you are using ?
Le sam. 28 sept. 2024, à 04 h 27, Duffer <diffd...@12post.net> a écrit : > On 2024-09-27 22:00, Duffer wrote: > > On 2024-09-27 05:29, Eric Lavarde wrote: > >> Hi, > >> > >> On 27/09/2024 07:09, Duffer wrote: > >>> On 2024-09-25 22:30, Duffer wrote: > >>>> Hi. > >>>> > >>>> I'm using rdiff-backup 2.2.6 for a large amount of data. All traffic > >>>> is encrypted via stunnel. > >>>> > >>>> I have two concerns: > >>>> > >>>> 1. Speed > >>>> > >>>> With SSH, the speed I've seen is 25-50 Mbps, probably averaging > >>>> 30-40 Mbps. iperf3 gets around 500 Mbps between my local system > >>>> (destination) and the remote system (source). The SSH speed seems > >>>> excessively slow to me - maybe 6-8% of the measured bandwidth. > >>>> > >>>> I switched to mapping the data source via NFS. Now I'm seeing full > >>>> speed for the transfers. > >>>> > >>>> Why is SSH so incredibly slow? Is that expected? I'm okay using NFS > >>>> but I liked the convenience of SSH. > >> > >> Well, I can only guess that it's the pickling and unpickling necessary > >> over the network. I'm working on improving performance but it takes > >> time. I first need to refactor and simplify the code. It could also be > >> worsened due to a weak CPU on one end or the other. Many small files > >> also increase a lot the overhead. > > > > Good to know. I think SSH has a lot of overhead, anyway. It's usually > > far from the fastest protocol. There's at least one project attempting > > to improve performance: https://www.psc.edu/hpn-ssh-home/ > > > >>>> 2. Regression > >>>> > >>>> As noted, my backup was very slow so I stopped it (CTRL-C). The next > >>>> time I ran the backup, it went into "regress" mode, so I lost all > >>>> the progress. Is there no way to avoid this and resume somehow? I > >>>> couldn't find anything in the FAQ. > >> > >> No, but there is already an issue for this. Again, takes time... > > > > Thanks. Yes, it's definitely something you want to get right before > > deploying it. > > > >>> The regress didn't take long at all but it's now been running over 10 > >>> hours and I've seen little progress. Disk usage hasn't increased at > >>> all. Is this expected? I thought after the regress it would start > >>> downloading, like it did before. I can't tell what it's doing but I > >>> seem to see it reading a lot from the disk. I used "-v 3" but maybe I > >>> should have opted for more output. > >> > >> -v3 is the default value, I generally use -v5 to see progress (you can > >> configure differently logfile and console output BTW). > >> Difficult to say what's happening from here, but did you check if > >> files are appearing in your backup repo, especially under > >> rdiff-backup-data? > >> If everything breaks, assuming Linux, you can try to guess what it's > >> doing using strace. > >> > >> KR, Eric > > > > I'm seeing continuous creation of files like rdiff-backup.tmp.217452 in > > the backup path. There are also lots of files under rdiff-backup-data, > > including two one file and one directory from today. > > > > It's been running 18 hours, so I hate to break it again but I really > > wish I could figure out what it's doing. One of the backup directories > > (where I see the tmp files) keeps growing and shrinking. > > It's been running almost 24 hours now. I see lots of activity, accessing > files in the actual data, .diff.gz files, and .tmp files. But the disk > usage still hasn't changed. Is it possible it's re-verifying every > single file? I wish I could figure out what it is doing. This seems > completely different from the previous run that I stopped. > > -- IKUS Software https://www.ikus-soft.com/ 514-971-6442 St-Colomban, QC J5K 1T9
