Re: --compare-dest weirdness
On 21.07.2010 10:22, Kevin Murray wrote: Hi Henri Thanks very much. LBackup looks good, but i need a solution which only copies files which have changed, and does not link the others, as the folder is destined for a tar archive. Also, the copied files must be fully functional, i.e. must be whole files, not just differences like rdiff. this is why im using rsync. Using checksum works, however with over 600gb to backup it is not a viable option. That's easy: rsync with --link-dest so that all files that aren't changed are hardlinked. Then to 'tar' all changed or new files you want a list of files without hardlinks as those are the files that where changed or new(*): find dir -type f -links 1 files.lst tar options --files-from=files.lst *: I'm assuming hardlinks aren't used within the source-directory. Otherwise find would skip those files too as they would also have more than 1 link. Also symlinks and special-files are excluded by the example. Bis denn -- Real Programmers consider what you see is what you get to be just as bad a concept in Text Editors as it is in women. No, the Real Programmer wants a you asked for it, you got it text editor -- complicated, cryptic, powerful, unforgiving, dangerous. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --compare-dest weirdness
On Wed, 21 Jul 2010, Matthias Schniedermeyer wrote: rsync with --link-dest so that all files that aren't changed are hardlinked. Then to 'tar' all changed or new files you want a list of files without hardlinks as those are the files that where changed or new(*): find dir -type f -links 1 files.lst tar options --files-from=files.lst Easier yet: rsnapshot -z (use compression) Simple way to get versions - hoursly/daily/weekly/monthly. Lee -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --compare-dest weirdness
Hi Henri Thanks very much. LBackup looks good, but i need a solution which only copies files which have changed, and does not link the others, as the folder is destined for a tar archive. Also, the copied files must be fully functional, i.e. must be whole files, not just differences like rdiff. this is why im using rsync. Using checksum works, however with over 600gb to backup it is not a viable option. the -i shows that most are being copied due to time differences, so in theory -t should work? This does in fact work on a little test setup on my work laptop, i will test it properly when i get home tonight. Thanks Regards Kevin Murray On 20 July 2010 12:40, Henri Shustak henri.shus...@gmail.com wrote: interestingly, i tried to see if something was wrong with my statments by doing: mkdir ~/rsynctest/dir1 mkdir ~/rsynctest/dir2 mkdir ~/rsynctest/dir3 nano ~/rsynctest/dir1/file1 (wrote the line hello world and saved) nano ~/rsynctest/dir1/file2 (wrote the line hello and saved) cp ~/rsynctest/dir1/file* ~/rsynctest/dir2/ checked md5sums of both files in both dirs to ensure they were identical the did rsync -rvu --compare-dest=/home/kevin/rsynctest/dir2/ /home/kevin/rsynctest/dir1/ /home/kevin/rsynctest/dir3/ the two files from dir1 were copied to dir3, even though identical copies, with the same names, existed in dir2. i also tried this again with the compare dir relative to the dest dir, i.e.: rsync -rvu --compare-dest=../dir2/ /home/kevin/rsynctest/dir1/ /home/kevin/rsynctest/dir3/ again with same result I thought about this further and I think the problem you are having is that the modification times are not being preserved and you are not using the --checksum option. As such, I suggest that you add the --times or --checksum option and see if that helps. I suspect that this will solve your problem. Let me know =:^) - This email is protected by LBackup, an open source backup solution. Free as in freedom; LBackup is licensed under the GNU GPL http://www.lbackup.org -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --compare-dest weirdness
the -i shows that most are being copied due to time differences, so in theory -t should work? This does in fact work on a little test setup on my work laptop, i will test it properly when i get home tonight. I am glad that the -t option is working with your test. If you decide to use LBackup and have a full hard linked backup then a possible option is to have a post action script which passes a list of files and directories which have been updated or created to tar for archive creation. To enable rsync session logging add the following line to the LBackup backup configuration file : enable_rsync_session_log=YES Then using tar with the -n option you would be able to feed in a list of newly created or modified files. Depending upon what kind of information you want to preserve something like the following could do the trick. However, please keep in mind that you may need to add or remove tar various options. tar -c -n -z -v -f /tmp/new_archive.tar.gz -T /tmp/list_to_archive.txt If you opted for such an approach you could potentially stop using the --compare-dest option and rely upon the output from rsync for a list of files you would like to archive. All the best with your backup script. With any luck the rsync -t option will work with you backup as well as it did for your testing. --- This email is protected by LBackup, an open source backup solution. http://www.lbackup.org -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --compare-dest weirdness
interestingly, i tried to see if something was wrong with my statments by doing: mkdir ~/rsynctest/dir1 mkdir ~/rsynctest/dir2 mkdir ~/rsynctest/dir3 nano ~/rsynctest/dir1/file1 (wrote the line hello world and saved) nano ~/rsynctest/dir1/file2 (wrote the line hello and saved) cp ~/rsynctest/dir1/file* ~/rsynctest/dir2/ checked md5sums of both files in both dirs to ensure they were identical the did rsync -rvu --compare-dest=/home/kevin/rsynctest/dir2/ /home/kevin/rsynctest/dir1/ /home/kevin/rsynctest/dir3/ the two files from dir1 were copied to dir3, even though identical copies, with the same names, existed in dir2. i also tried this again with the compare dir relative to the dest dir, i.e.: rsync -rvu --compare-dest=../dir2/ /home/kevin/rsynctest/dir1/ /home/kevin/rsynctest/dir3/ again with same result I thought about this further and I think the problem you are having is that the modification times are not being preserved and you are not using the --checksum option. As such, I suggest that you add the --times or --checksum option and see if that helps. I suspect that this will solve your problem. Let me know =:^) - This email is protected by LBackup, an open source backup solution. Free as in freedom; LBackup is licensed under the GNU GPL http://www.lbackup.org -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--compare-dest weirdness
Hi All, I am writing a backup program for my computer. brief outline is as follows. Running ubuntu 10.04 2 main partitions, / and /home, both ext3. 1 external usb hdd, ext3, mounted to /backups/main. once every couple of days, rsync backs up, using following command, everything worth backing up in / and /home partitions to a folder /backups/main/Full. command: rsync -vrhRupElog --delete-during --delete-excluded --exclude-from=/backups/.config/full.exclude --log-file=/backups/.logs/full.rlog / /backups/main/Full/. this works perfectly now what i want to do, is every few hours, rsync ONLY the files changed in the last few days to a folder in /backups/main/Incremental/MMddTHHmm, where the ... is a string representing the date and time. (and then tar.gz the previous directory to save space) the command i am using for this is : rsync -vrhRupElog --delete-during --delete-excluded --exclude-from=/backups/.config/incremental.exclude --log-file=/backups/.logs/incremental.rlog --compare-dest=/backups/main/Full/ / /backups/main/Incremental/Full/20100716T1355 the incremental.exclude differs from the full.exclude only in that i also exclude .mp3, .avi and other large and non-critical file types. the problem is that rsync is behaving as though it is ignoring the compare-dest option. it throws no error about this dir not existing, and i have tried giving it as a relative dir to the destination with same result. interestingly, i tried to see if something was wrong with my statments by doing: mkdir ~/rsynctest/dir1 mkdir ~/rsynctest/dir2 mkdir ~/rsynctest/dir3 nano ~/rsynctest/dir1/file1 (wrote the line hello world and saved) nano ~/rsynctest/dir1/file2 (wrote the line hello and saved) cp ~/rsynctest/dir1/file* ~/rsynctest/dir2/ checked md5sums of both files in both dirs to ensure they were identical the did rsync -rvu --compare-dest=/home/kevin/rsynctest/dir2/ /home/kevin/rsynctest/dir1/ /home/kevin/rsynctest/dir3/ the two files from dir1 were copied to dir3, even though identical copies, with the same names, existed in dir2. i also tried this again with the compare dir relative to the dest dir, i.e.: rsync -rvu --compare-dest=../dir2/ /home/kevin/rsynctest/dir1/ /home/kevin/rsynctest/dir3/ again with same result is this a bug, or have i got the wrong end of the stick regarding --compare-dir? thanks all. Regards Kevin Murray full.exclude Description: Binary data incremental.exclude Description: Binary data -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
