Re: cross-platform backup tool incremental rdiff-backup takes long
Thank you for your fast reply! Deleting a file is not a simple operation. Since rdiff-backup always works backward from the current mirror, restoring a file as of some previous date requires storing a snapshot of the last known state of that file in the increments. By default, those snapshots are compressed except for file names that match the --no-compression-regexp (see Globals.py for the default expression), and that compression can take a lot of time. This makes totally sense! You might want to take a look at the "--no-compression" option in the manpage. I will have a look at this option. Cheers, Jonas
Re: cross-platform backup tool incremental rdiff-backup takes long
Hi all! I am using rdiff-backup without flaws since this conversation here last year. Although I always need to add the "--no-fsync" argument in order to get fast (20-30min) backups. Unfortunately, I have a problem with the current backup. I deleted a lot of stuff on my machine (around 200GB) and if I do a new backup I have to abort it after 9 hours as it does not finish. iostat -x -m 3 shows that %util is always between 70 and 95. So I assume that IO is again a bottleneck, but shouldn't it be easy for rdiff-backup to just delete the files from the backup? Is there a way how I can speed up this process? Thank you very much and cheers, Jonas On 6/2/21 13:18, Patrik Dufresne wrote: Hi Jonas, Great news! Fsync is useful but not in you scenario. I'm glad it worked for you. Don't hesitate to come back of you have other questions.
Re: cross-platform backup tool rdiff-backup on android phone
Hi Leland! No worries about responses :) I mount my Android device on my laptop and then start the backup of the mounted folder. So such a loop cannot be a problem and at the moment I have no clue what could cause this. If I find the time then I will re-try and test it on the weekend again. Cheers, Jonas On 1/26/22 20:44, Leland Best wrote: Hi Jonas, Apologies for the slow response. Some neurological issues plus Real Life make it hard to keep up with e-mail. :( Anyway, ... On Sun, 2022-01-23 at 19:44 +0100, Jonas Schöpf wrote: Hi Leland! Thank you for this suggestion. You're welcome. Mounting with sshfs works just fine. I have tried a backup with it, but rdiff-backup is looping when using sshfs. Does anybody have an idea why? So I mean after everything is copied from my phone it starts again copying from the beginning. [...] Hmmm. Strange. Although, as I said, I've never done this myself. You aren't by any chance mounting _both_ ways with 'sshfs' are you? That is, mounting the phone on the backup server, _and_ mounting the backup server on the phone? Seems like that would create a recursive loop ... ? Otherwise, when time permits, I can try some tests between two Linux machines (sadly, my phone isn't set up for 'ssh' and friends). Cheers Leland -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: cross-platform backup tool rdiff-backup on android phone
Hi Leland! Thank you for this suggestion. Mounting with sshfs works just fine. I have tried a backup with it, but rdiff-backup is looping when using sshfs. Does anybody have an idea why? So I mean after everything is copied from my phone it starts again copying from the beginning. Cheers, Jonas On 1/12/22 02:04, Leland Best wrote: Hi Jonas, On Sun, 2022-01-09 at 17:31 +0100, Jonas Schöpf wrote: Hi all! I have question about using rdiff-backup to back up my android phone. The backup would be done over ssh, but the problem is that I am (as far as I know) not able to install rdiff-backup on my phone. Is there still a possibility to use rdiff-backup from my PC to create a backup of my android phone via ssh? If you can 'ssh' to your phone, can you install some "flavor" of 'sshfs'? If so, then it seems like you could mount your phone over 'sshfs' on a machine with 'rdiff-backup' installed, then do what looks like a backup of a local directory (but is really the 'sshfs' mount point). Mind you, I've never tried this. Just a thought. I _have_ used 'exfat' over USB but that has serious issues with file metadata (date, time, etc.). Thank you and cheers, Jonas Cheers Leland -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: cross-platform backup tool rdiff-backup on android phone
Sorry, yes I forgot to mention that I used simple-mtpfs. I did not have the time yet to test it via ssh. On 1/11/22 14:05, EricZolf wrote: But you used mtpfs? On 11 January 2022 08:03:11 UTC, "Jonas Schöpf" wrote: Hi all! The backup ran with <10 errors (some files could not be copied), but else it was fine! Thanks again! On 1/10/22 14:30, Jonas Schöpf wrote: Hi! @Eric: Thanks for this interesting pointer. If I find the time, I will definitely try it and report back. @Derek: Thank you very much. For now I will use simple-mtpfs for doing a backup. At the moment the backup is running with any errors. Cheers, Jonas On 1/10/22 13:57, Derek Atkins wrote: Hi, On Sun, January 9, 2022 11:31 am, Jonas Schöpf wrote: Hi all! I have question about using rdiff-backup to back up my android phone. The backup would be done over ssh, but the problem is that I am (as far as I know) not able to install rdiff-backup on my phone. Is there still a possibility to use rdiff-backup from my PC to create a backup of my android phone via ssh? I don't think you'll be able to do that. I mount my android phone using simple-mtpfs and then using rdiff-backup from there. Thank you and cheers, Jonas -derek -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: cross-platform backup tool rdiff-backup on android phone
Hi all! The backup ran with <10 errors (some files could not be copied), but else it was fine! Thanks again! On 1/10/22 14:30, Jonas Schöpf wrote: Hi! @Eric: Thanks for this interesting pointer. If I find the time, I will definitely try it and report back. @Derek: Thank you very much. For now I will use simple-mtpfs for doing a backup. At the moment the backup is running with any errors. Cheers, Jonas On 1/10/22 13:57, Derek Atkins wrote: Hi, On Sun, January 9, 2022 11:31 am, Jonas Schöpf wrote: Hi all! I have question about using rdiff-backup to back up my android phone. The backup would be done over ssh, but the problem is that I am (as far as I know) not able to install rdiff-backup on my phone. Is there still a possibility to use rdiff-backup from my PC to create a backup of my android phone via ssh? I don't think you'll be able to do that. I mount my android phone using simple-mtpfs and then using rdiff-backup from there. Thank you and cheers, Jonas -derek -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: cross-platform backup tool rdiff-backup on android phone
Hi! @Eric: Thanks for this interesting pointer. If I find the time, I will definitely try it and report back. @Derek: Thank you very much. For now I will use simple-mtpfs for doing a backup. At the moment the backup is running with any errors. Cheers, Jonas On 1/10/22 13:57, Derek Atkins wrote: Hi, On Sun, January 9, 2022 11:31 am, Jonas Schöpf wrote: Hi all! I have question about using rdiff-backup to back up my android phone. The backup would be done over ssh, but the problem is that I am (as far as I know) not able to install rdiff-backup on my phone. Is there still a possibility to use rdiff-backup from my PC to create a backup of my android phone via ssh? I don't think you'll be able to do that. I mount my android phone using simple-mtpfs and then using rdiff-backup from there. Thank you and cheers, Jonas -derek -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
cross-platform backup tool rdiff-backup on android phone
Hi all! I have question about using rdiff-backup to back up my android phone. The backup would be done over ssh, but the problem is that I am (as far as I know) not able to install rdiff-backup on my phone. Is there still a possibility to use rdiff-backup from my PC to create a backup of my android phone via ssh? Thank you and cheers, Jonas
Re: cross-platform backup tool Lack of reviewers (again) and decision
Hi Eric! First of all, thank you for all the work you put into rdiff-backup! I love this tool! Last time you wrote an email about lack of reviewers, I took a look at the PRs, but couldn't decide anything about them as I lack knowledge of the implementation. I "watch" the repo now and try to find some time to look at the code such that in the future I might be somehow more helpful for this project :) Cheers, Jonas On 11/10/21 06:50, Eric L. Zolf wrote: Hi, I don't want to sound like a broken record but I have again a pull request [1] waiting for review and nobody takes care. Our contribution guide states that "Ideally each pull request gets some feedback within 24 hours from it having been filed" and we're far from it. I'm also tired of begging for review. For this reason, I'll stop begging and will merge my own PRs after a little while (24h-48h, perhaps more if the PR isn't blocking me). Not exactly development best practice, but anybody disagreeing is free to commit to provide timely feedback to PRs, and I'll happily support them in this endeavor. Thanks for your attention, Eric [1] https://github.com/rdiff-backup/rdiff-backup/pull/632
Re: incremental rdiff-backup takes long
Hi! Thank you again for your help! With the "--no-fsync" argument, the backup succeeded after 13 minutes. Cheers, Jonas On 6/1/21 9:53 PM, Jonas Schöpf wrote: > Hi Patrik! > > On 6/1/21 4:48 PM, Patrik Dufresne wrote: >> Hello Jonas, >> >> Welcome to rdiff-backup ! Let me try to help you a bit more with you >> problem. > Thanks :) >> >> First, the first backup always take alot of time because rdiff-backup is >> not like a simple copy and alot of overhead are done in the background >> to put in place the first version of the data. >> >> Second, with your use case of 400GB, I would not expect a very long >> elapsed time for subsequent backup. I have a couple of backups with >> similar size and it roughly takes ~10-20 minutes over the Internet with >> 400Mbps. But you might not have the same setup as I do. >> >> If $HOME and $DRIVE are both local, It's quite simple to narrow the problem. >> >> While running the backup with you command line, you might want to check >> how the IO of your computer behaves. In the command line, try executring >> `iostat -x -m 3` and take a look at the '%util' column. It displays the >> percentage utilisation of the disk. If the value is near 90%, you have >> found the culprit. The IO is your bottleneck. >> >> To help the situation, you might want to run the backup without fsync. >> When Fsync is enabled, it forces the system to wait until the data is >> written to the disk and persisted. For slow drives on USB, it may have a >> huge impact on the performance. So try running rdiff-backup with >> `--no-fsync`. > > Thank you for your help. > Sounds like this is the problem! > > I will try your this tomorrow and report back if it works :) > > Thank you all for your help! > > Cheers, > Jonas > > -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: incremental rdiff-backup takes long
Hi Patrik! On 6/1/21 4:48 PM, Patrik Dufresne wrote: > Hello Jonas, > > Welcome to rdiff-backup ! Let me try to help you a bit more with you > problem. Thanks :) > > First, the first backup always take alot of time because rdiff-backup is > not like a simple copy and alot of overhead are done in the background > to put in place the first version of the data. > > Second, with your use case of 400GB, I would not expect a very long > elapsed time for subsequent backup. I have a couple of backups with > similar size and it roughly takes ~10-20 minutes over the Internet with > 400Mbps. But you might not have the same setup as I do. > > If $HOME and $DRIVE are both local, It's quite simple to narrow the problem. > > While running the backup with you command line, you might want to check > how the IO of your computer behaves. In the command line, try executring > `iostat -x -m 3` and take a look at the '%util' column. It displays the > percentage utilisation of the disk. If the value is near 90%, you have > found the culprit. The IO is your bottleneck. > > To help the situation, you might want to run the backup without fsync. > When Fsync is enabled, it forces the system to wait until the data is > written to the disk and persisted. For slow drives on USB, it may have a > huge impact on the performance. So try running rdiff-backup with > `--no-fsync`. Thank you for your help. Sounds like this is the problem! I will try your this tomorrow and report back if it works :) Thank you all for your help! Cheers, Jonas -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: incremental rdiff-backup takes long
On 6/1/21 4:20 PM, Yves Bellefeuille wrote: > "Jonas Schoepf" > >> My drive is connected via USB. > > I think we're on the right track. What version of USB? I have a USB 3.2 Gen 1. > >> Before using rdiff-backup I used just rsync, where the initial backup >> took also quite long, but the following backups took ~20 minutes. > > rdiff-backup will always be slower than rsync, since rdiff-backup > keeps previous versions, unlike rsync. Yes, I understand that. But the difference can't be 20 minutes vs. more than 8 hours... - Jonas -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: incremental rdiff-backup takes long
My drive is connected via USB. Before using rdiff-backup I used just rsync, where the initial backup took also quite long, but the following backups took ~20 minutes. - Jonas On 6/1/21 4:03 PM, Yves Bellefeuille wrote: > Jonas Schoepf wrote: > >> I understand that the initial backup is really slow and took more >> than ~10h for my 400 GB. > > > I'd suspect slow reading or writing speeds. How are the drives > connected (SATA, USB, network, etc.)? > -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
Re: incremental rdiff-backup takes long
Thank you for the quick response! The current backup runs since 6h and it only uses around 1-2% of the CPU. There is also no gzip process running. So it might be something else... - Jonas On 6/1/21 3:15 PM, Reio Remma wrote: > On 01.06.2021 16:12, Jonas Schöpf wrote: >> Hi all! >> >> I found out about rdiff-backup two weeks ago after I searched for a >> method to get incremental backups of my system. >> I experience some problems while doing my backup and I hope that >> somebody can tell me what I am doing wrong. >> >> I understand that the initial backup is really slow and took more than >> ~10h for my 400 GB. >> Now if I do another backup, I would expect that it is quite fast, but >> after it exceeded 8h and I just aborted the whole backup process. >> >> I use the following command: >>> rdiff-backup --terminal-verbosity 8 --print-statistics \ >>> --exclude-globbing-filelist $HOME/.excludes \ >>> $HOME $DRIVE >> Can anybody tell me what I am doing wrong? >> >> Thank you for your help! >> >> Cheers, >> Jonas > > Hello! A while ago I had a similar problem and I narrowed it down to > rdiff-backup compressing files that were already compressed. I would > suggest having a look at CPU usage etc when the backup is running. iirc > it uses gzip which is fairly processor hungry. > > Good luck! > Reio > -- GPG fingerprint: 0BF3 B30B F1D5 6556 795E F68F 8626 F794 FE62 BE1F
incremental rdiff-backup takes long
Hi all! I found out about rdiff-backup two weeks ago after I searched for a method to get incremental backups of my system. I experience some problems while doing my backup and I hope that somebody can tell me what I am doing wrong. I understand that the initial backup is really slow and took more than ~10h for my 400 GB. Now if I do another backup, I would expect that it is quite fast, but after it exceeded 8h and I just aborted the whole backup process. I use the following command: > rdiff-backup --terminal-verbosity 8 --print-statistics \ >--exclude-globbing-filelist $HOME/.excludes \ >$HOME $DRIVE Can anybody tell me what I am doing wrong? Thank you for your help! Cheers, Jonas
