Re: cross-platform backup tool incremental rdiff-backup takes long
Thank you for your fast reply! Deleting a file is not a simple operation. Since rdiff-backup always works backward from the current mirror, restoring a file as of some previous date requires storing a snapshot of the last known state of that file in the increments. By default, those snapshots are compressed except for file names that match the --no-compression-regexp (see Globals.py for the default expression), and that compression can take a lot of time. This makes totally sense! You might want to take a look at the "--no-compression" option in the manpage. I will have a look at this option. Cheers, Jonas
Re: cross-platform backup tool incremental rdiff-backup takes long
On 10/18/22 1:37 AM, Jonas Schöpf wrote: I deleted a lot of stuff on my machine (around 200GB) and if I do a new backup I have to abort it after 9 hours as it does not finish. iostat -x -m 3 shows that %util is always between 70 and 95. So I assume that IO is again a bottleneck, but shouldn't it be easy for rdiff-backup to just delete the files from the backup? Is there a way how I can speed up this process? Deleting a file is not a simple operation. Since rdiff-backup always works backward from the current mirror, restoring a file as of some previous date requires storing a snapshot of the last known state of that file in the increments. By default, those snapshots are compressed except for file names that match the --no-compression-regexp (see Globals.py for the default expression), and that compression can take a lot of time. You might want to take a look at the "--no-compression" option in the manpage. -- Bob Nichols "NOSPAM" is really part of my email address. Do NOT delete it.
Re: cross-platform backup tool incremental rdiff-backup takes long
Hi all! I am using rdiff-backup without flaws since this conversation here last year. Although I always need to add the "--no-fsync" argument in order to get fast (20-30min) backups. Unfortunately, I have a problem with the current backup. I deleted a lot of stuff on my machine (around 200GB) and if I do a new backup I have to abort it after 9 hours as it does not finish. iostat -x -m 3 shows that %util is always between 70 and 95. So I assume that IO is again a bottleneck, but shouldn't it be easy for rdiff-backup to just delete the files from the backup? Is there a way how I can speed up this process? Thank you very much and cheers, Jonas On 6/2/21 13:18, Patrik Dufresne wrote: Hi Jonas, Great news! Fsync is useful but not in you scenario. I'm glad it worked for you. Don't hesitate to come back of you have other questions.
Re: incremental rdiff-backup takes long
Hi Jonas, Great news! Fsync is useful but not in you scenario. I'm glad it worked for you. Don't hesitate to come back of you have other questions. On Wed., Jun. 2, 2021, 4:56 a.m. Jonas Schöpf, wrote: > Hi! > > Thank you again for your help! > > With the "--no-fsync" argument, the backup succeeded after 13 minutes. > > Cheers, > Jonas > > On 6/1/21 9:53 PM, Jonas Schöpf wrote: > > Hi Patrik! > > > > On 6/1/21 4:48 PM, Patrik Dufresne wrote: > >> Hello Jonas, > >> > >> Welcome to rdiff-backup ! Let me try to help you a bit more with you > >> problem. > > Thanks :) > >> > >> First, the first backup always take alot of time because rdiff-backup is > >> not like a simple copy and alot of overhead are done in the background > >> to put in place the first version of the data. > >> > >> Second, with your use case of 400GB, I would not expect a very long > >> elapsed time for subsequent backup. I have a couple of backups with > >> similar size and it roughly takes ~10-20 minutes over the Internet with > >> 400Mbps. But you might not have the same setup as I do. > >> > >> If $HOME and $DRIVE are both local, It's quite simple to narrow the > problem. > >> > >> While running the backup with you command line, you might want to check > >> how the IO of your computer behaves. In the command line, try executring > >> `iostat -x -m 3` and take a look at the '%util' column. It displays the > >> percentage utilisation of the disk. If the value is near 90%, you have > >> found the culprit. The IO is your bottleneck. > >> > >> To help the situation, you might want to run the backup without fsync. > >> When Fsync is enabled, it forces the system to wait until the data is > >> written to the disk and persisted. For slow drives on USB, it may have a > >> huge impact on the performance. So try running rdiff-backup with > >> `--no-fsync`. > > > > Thank you for your help. > > Sounds like this is the problem! > > > > I will try your this tomorrow and report back if it works :) > > > > Thank you all for your help! > > > > Cheers, > > Jonas > > > > > > -- > GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F > >
Re: incremental rdiff-backup takes long
Hi! Thank you again for your help! With the "--no-fsync" argument, the backup succeeded after 13 minutes. Cheers, Jonas On 6/1/21 9:53 PM, Jonas Schöpf wrote: > Hi Patrik! > > On 6/1/21 4:48 PM, Patrik Dufresne wrote: >> Hello Jonas, >> >> Welcome to rdiff-backup ! Let me try to help you a bit more with you >> problem. > Thanks :) >> >> First, the first backup always take alot of time because rdiff-backup is >> not like a simple copy and alot of overhead are done in the background >> to put in place the first version of the data. >> >> Second, with your use case of 400GB, I would not expect a very long >> elapsed time for subsequent backup. I have a couple of backups with >> similar size and it roughly takes ~10-20 minutes over the Internet with >> 400Mbps. But you might not have the same setup as I do. >> >> If $HOME and $DRIVE are both local, It's quite simple to narrow the problem. >> >> While running the backup with you command line, you might want to check >> how the IO of your computer behaves. In the command line, try executring >> `iostat -x -m 3` and take a look at the '%util' column. It displays the >> percentage utilisation of the disk. If the value is near 90%, you have >> found the culprit. The IO is your bottleneck. >> >> To help the situation, you might want to run the backup without fsync. >> When Fsync is enabled, it forces the system to wait until the data is >> written to the disk and persisted. For slow drives on USB, it may have a >> huge impact on the performance. So try running rdiff-backup with >> `--no-fsync`. > > Thank you for your help. > Sounds like this is the problem! > > I will try your this tomorrow and report back if it works :) > > Thank you all for your help! > > Cheers, > Jonas > > -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: incremental rdiff-backup takes long
Hi Patrik! On 6/1/21 4:48 PM, Patrik Dufresne wrote: > Hello Jonas, > > Welcome to rdiff-backup ! Let me try to help you a bit more with you > problem. Thanks :) > > First, the first backup always take alot of time because rdiff-backup is > not like a simple copy and alot of overhead are done in the background > to put in place the first version of the data. > > Second, with your use case of 400GB, I would not expect a very long > elapsed time for subsequent backup. I have a couple of backups with > similar size and it roughly takes ~10-20 minutes over the Internet with > 400Mbps. But you might not have the same setup as I do. > > If $HOME and $DRIVE are both local, It's quite simple to narrow the problem. > > While running the backup with you command line, you might want to check > how the IO of your computer behaves. In the command line, try executring > `iostat -x -m 3` and take a look at the '%util' column. It displays the > percentage utilisation of the disk. If the value is near 90%, you have > found the culprit. The IO is your bottleneck. > > To help the situation, you might want to run the backup without fsync. > When Fsync is enabled, it forces the system to wait until the data is > written to the disk and persisted. For slow drives on USB, it may have a > huge impact on the performance. So try running rdiff-backup with > `--no-fsync`. Thank you for your help. Sounds like this is the problem! I will try your this tomorrow and report back if it works :) Thank you all for your help! Cheers, Jonas -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: incremental rdiff-backup takes long
On 6/1/21 4:20 PM, Yves Bellefeuille wrote: > "Jonas Schoepf" > >> My drive is connected via USB. > > I think we're on the right track. What version of USB? I have a USB 3.2 Gen 1. > >> Before using rdiff-backup I used just rsync, where the initial backup >> took also quite long, but the following backups took ~20 minutes. > > rdiff-backup will always be slower than rsync, since rdiff-backup > keeps previous versions, unlike rsync. Yes, I understand that. But the difference can't be 20 minutes vs. more than 8 hours... - Jonas -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: incremental rdiff-backup takes long
On 2021-06-01 10:32 a.m., Yves Bellefeuille wrote: Alvin Starr wrote: I thought rdiff-backup like rsync keeps track of the file metadata like size and creation date to skip reading the data again if the file has not been updated? I assume rdiff-backup does the same thing to decide if a file has changed, but if there's a change, rsync simply throws away the old version. rdiff-backup notes the changes and keeps older versions (depending on the options) so you're not limited to only one previous version. That's the whole point. That may be the issue then. If the first run takes as long as subsequent runs then something may be interfering with the file checks used to avoid reading the data for comparison. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net ||
Re: incremental rdiff-backup takes long
Hello Jonas, Welcome to rdiff-backup ! Let me try to help you a bit more with you problem. First, the first backup always take alot of time because rdiff-backup is not like a simple copy and alot of overhead are done in the background to put in place the first version of the data. Second, with your use case of 400GB, I would not expect a very long elapsed time for subsequent backup. I have a couple of backups with similar size and it roughly takes ~10-20 minutes over the Internet with 400Mbps. But you might not have the same setup as I do. If $HOME and $DRIVE are both local, It's quite simple to narrow the problem. While running the backup with you command line, you might want to check how the IO of your computer behaves. In the command line, try executring `iostat -x -m 3` and take a look at the '%util' column. It displays the percentage utilisation of the disk. If the value is near 90%, you have found the culprit. The IO is your bottleneck. To help the situation, you might want to run the backup without fsync. When Fsync is enabled, it forces the system to wait until the data is written to the disk and persisted. For slow drives on USB, it may have a huge impact on the performance. So try running rdiff-backup with `--no-fsync`. On Tue, Jun 1, 2021 at 10:33 AM Yves Bellefeuille wrote: > Alvin Starr wrote: > > > I thought rdiff-backup like rsync keeps track of the file metadata > > like size and creation date to skip reading the data again if the > > file has not been updated? > > I assume rdiff-backup does the same thing to decide if a file has > changed, but if there's a change, rsync simply throws away the old > version. > > rdiff-backup notes the changes and keeps older versions (depending on > the options) so you're not limited to only one previous version. > That's the whole point. > > -- > Yves Bellefeuille > > > > > -- IKUS Software inc. https://www.ikus-soft.com/ 514-971-6442 130 rue Doris St-Colomban, QC J5K 1T9
Re: incremental rdiff-backup takes long
Alvin Starr wrote: > I thought rdiff-backup like rsync keeps track of the file metadata > like size and creation date to skip reading the data again if the > file has not been updated? I assume rdiff-backup does the same thing to decide if a file has changed, but if there's a change, rsync simply throws away the old version. rdiff-backup notes the changes and keeps older versions (depending on the options) so you're not limited to only one previous version. That's the whole point. -- Yves Bellefeuille
Re: incremental rdiff-backup takes long
On 2021-06-01 10:20 a.m., Yves Bellefeuille wrote: "Jonas Schoepf" My drive is connected via USB. I think we're on the right track. What version of USB? Before using rdiff-backup I used just rsync, where the initial backup took also quite long, but the following backups took ~20 minutes. rdiff-backup will always be slower than rsync, since rdiff-backup keeps previous versions, unlike rsync. I thought rdiff-backup like rsync keeps track of the file metadata like size and creation date to skip reading the data again if the file has not been updated? That should cause the subsequent runs to be much faster even with slow disks. -- Alvin Starr || land: (647)478-6285 Netvel Inc. || Cell: (416)806-0133 al...@netvel.net ||
Re: incremental rdiff-backup takes long
"Jonas Schoepf" > My drive is connected via USB. I think we're on the right track. What version of USB? > Before using rdiff-backup I used just rsync, where the initial backup > took also quite long, but the following backups took ~20 minutes. rdiff-backup will always be slower than rsync, since rdiff-backup keeps previous versions, unlike rsync. -- Yves Bellefeuille
Re: incremental rdiff-backup takes long
My drive is connected via USB. Before using rdiff-backup I used just rsync, where the initial backup took also quite long, but the following backups took ~20 minutes. - Jonas On 6/1/21 4:03 PM, Yves Bellefeuille wrote: > Jonas Schoepf wrote: > >> I understand that the initial backup is really slow and took more >> than ~10h for my 400 GB. > > > I'd suspect slow reading or writing speeds. How are the drives > connected (SATA, USB, network, etc.)? > -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: incremental rdiff-backup takes long
Jonas Schoepf wrote: > I understand that the initial backup is really slow and took more > than ~10h for my 400 GB. I'd suspect slow reading or writing speeds. How are the drives connected (SATA, USB, network, etc.)? -- Yves Bellefeuille
Re: incremental rdiff-backup takes long
Thank you for the quick response! The current backup runs since 6h and it only uses around 1-2% of the CPU. There is also no gzip process running. So it might be something else... - Jonas On 6/1/21 3:15 PM, Reio Remma wrote: > On 01.06.2021 16:12, Jonas Schöpf wrote: >> Hi all! >> >> I found out about rdiff-backup two weeks ago after I searched for a >> method to get incremental backups of my system. >> I experience some problems while doing my backup and I hope that >> somebody can tell me what I am doing wrong. >> >> I understand that the initial backup is really slow and took more than >> ~10h for my 400 GB. >> Now if I do another backup, I would expect that it is quite fast, but >> after it exceeded 8h and I just aborted the whole backup process. >> >> I use the following command: >>> rdiff-backup --terminal-verbosity 8 --print-statistics \ >>> --exclude-globbing-filelist $HOME/.excludes \ >>> $HOME $DRIVE >> Can anybody tell me what I am doing wrong? >> >> Thank you for your help! >> >> Cheers, >> Jonas > > Hello! A while ago I had a similar problem and I narrowed it down to > rdiff-backup compressing files that were already compressed. I would > suggest having a look at CPU usage etc when the backup is running. iirc > it uses gzip which is fairly processor hungry. > > Good luck! > Reio > -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: incremental rdiff-backup takes long
On 01.06.2021 16:12, Jonas Schöpf wrote: Hi all! I found out about rdiff-backup two weeks ago after I searched for a method to get incremental backups of my system. I experience some problems while doing my backup and I hope that somebody can tell me what I am doing wrong. I understand that the initial backup is really slow and took more than ~10h for my 400 GB. Now if I do another backup, I would expect that it is quite fast, but after it exceeded 8h and I just aborted the whole backup process. I use the following command: rdiff-backup --terminal-verbosity 8 --print-statistics \ --exclude-globbing-filelist $HOME/.excludes \ $HOME $DRIVE Can anybody tell me what I am doing wrong? Thank you for your help! Cheers, Jonas Hello! A while ago I had a similar problem and I narrowed it down to rdiff-backup compressing files that were already compressed. I would suggest having a look at CPU usage etc when the backup is running. iirc it uses gzip which is fairly processor hungry. Good luck! Reio
incremental rdiff-backup takes long
Hi all! I found out about rdiff-backup two weeks ago after I searched for a method to get incremental backups of my system. I experience some problems while doing my backup and I hope that somebody can tell me what I am doing wrong. I understand that the initial backup is really slow and took more than ~10h for my 400 GB. Now if I do another backup, I would expect that it is quite fast, but after it exceeded 8h and I just aborted the whole backup process. I use the following command: > rdiff-backup --terminal-verbosity 8 --print-statistics \ >--exclude-globbing-filelist $HOME/.excludes \ >$HOME $DRIVE Can anybody tell me what I am doing wrong? Thank you for your help! Cheers, Jonas
