Re: With rsync --link-dest, is it possible to determine which files are new?
SHA1 You may also find fingerprint to be a useful tool. However, it will require ruby. --- This email is protected by LBackup http://www.lbackup.org -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
On 10/21/2011 07:14 PM, Ido Magal wrote: Thanks! Unfortunately, it doesn't look like the perl for DroboFS supports the required libraries for your script, error while loading shared libraries: libperl.so.5.10: cannot open shared object file: No such file or directory but --itemize-changes should be sufficient. On Fri, Oct 21, 2011 at 09:10, Kevin Korbk...@sanitarium.net wrote: --itemize-changes # find /backup/dir -type f -links 1 ? -- -Eric 'shubes' -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
On Fri, 21 Oct 2011 12:10:09 -0400, Kevin Korb wrote: If you want something you can run after the fact here is a tool I wrote a while back that does a sort of diff across 2 --link-dest based backups: http://sanitarium.net/unix_stuff/rspaghetti_backup/diff_backup.pl.txt It will also tell you what files were not included in the newer backup which --itemize-changes will not since it doesn't actually --delete anything. I like how comprehensive this is, but wouldn't it - by definition of what it means to have a new file in the new directory tree - always report a modified file as having a different inode? That takes me a quick and dirty solution for this. Simply do a find looking for files with a -links (link count) of 1 to see what files are newly copied in the newer directory tree. - Andrew -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yes, if a file is a different inode it will be reported by my script even if that is the only difference. If the inode number is different then the file is in fact different and takes up additional disk space. If that is the only thing that is different you might be using rsync incorrectly. This is also why I use md5sum to check files that have different inode numbers but the same mtime and file size. On 10/22/11 10:12, Andrew Gideon wrote: On Fri, 21 Oct 2011 12:10:09 -0400, Kevin Korb wrote: If you want something you can run after the fact here is a tool I wrote a while back that does a sort of diff across 2 --link-dest based backups: http://sanitarium.net/unix_stuff/rspaghetti_backup/diff_backup.pl.txt It will also tell you what files were not included in the newer backup which --itemize-changes will not since it doesn't actually --delete anything. I like how comprehensive this is, but wouldn't it - by definition of what it means to have a new file in the new directory tree - always report a modified file as having a different inode? That takes me a quick and dirty solution for this. Simply do a find looking for files with a -links (link count) of 1 to see what files are newly copied in the newer directory tree. - Andrew - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6i6kUACgkQVKC1jlbQAQfmXACglTh5+UXDdTSFEDZlmWnauUSh 3NEAni0s5XDg0ntixrdNWy+7nogIW9+r =/7b8 -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Opps, forgot that one. That isn't a perl library either but another shell command. On 10/22/11 11:02, Eric Shubert wrote: On 10/21/2011 07:14 PM, Ido Magal wrote: Thanks! Unfortunately, it doesn't look like the perl for DroboFS supports the required libraries for your script, error while loading shared libraries: libperl.so.5.10: cannot open shared object file: No such file or directory but --itemize-changes should be sufficient. On Fri, Oct 21, 2011 at 09:10, Kevin Korbk...@sanitarium.net wrote: --itemize-changes # find /backup/dir -type f -links 1 ? - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6i6mwACgkQVKC1jlbQAQcRRQCgs4p35+qoNH7OsuzoPMaL3Jee 2AYAoMAF/EeLQ6rJsRZuL5qst0PIYesF =p81Y -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
On Sat, 22 Oct 2011 12:07:33 -0400, Kevin Korb wrote: If that is the only thing that is different you might be using rsync incorrectly. And that's why you left it being reported? Interesting idea. Thanks for explaining. - Andrew -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
On Fri, 21 Oct 2011 19:14:00 -0700, Ido Magal wrote: error while loading shared libraries: libperl.so.5.10: cannot open shared object file: No such file or directory This doesn't appear to be a complaint about something the Perl script is doing, but about Perl itself not working. - Andrew -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Well, I had 2 reasons for writing the script... 1. I wanted to know what was different between 2 backups. Capturing - --itemize-changes can certainly do that (I don't remember if my script pre-dates --itemize-changes or not but it predates my knowledge of it) 2. I wanted to know what files were unique to a backup and therefore consuming disk space and why they were different. If I saw an instance where the only difference was the inode number then I would be concerned and investigate why rsync chose to duplicate that file. I have never actually witnessed this happening but I only checked for it a few times. On 10/22/11 15:17, Andrew Gideon wrote: On Sat, 22 Oct 2011 12:07:33 -0400, Kevin Korb wrote: If that is the only thing that is different you might be using rsync incorrectly. And that's why you left it being reported? Interesting idea. Thanks for explaining. - Andrew - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6jGloACgkQVKC1jlbQAQevDQCg5Kh5UyR8ymHgXZmlmcSwmNkK kXIAoJzpMbI8aYZw/OJ9iQEIpee+xpcK =QeCn -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
'find' on the droboFS doesn't support '-links' so I had to get creative. And apparently perl has issues on it as well. On Sat, Oct 22, 2011 at 08:02, Eric Shubert e...@shubes.net wrote: # find /backup/dir -type f -links 1 ? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
On 10/22/2011 04:04 PM, Ido Magal wrote: 'find' on the droboFS doesn't support '-links' so I had to get creative. And apparently perl has issues on it as well. On Sat, Oct 22, 2011 at 08:02, Eric Shuberte...@shubes.net wrote: # find /backup/dir -type f -links 1 ? Sorry, I'm not familiar with droboFS. So I checked it out a bit. From the looks if things, droboFS may not (from their web site) deliver the best file sharing experience ever. I certainly wouldn't trust my data to BeyondRAID technology alone (give me software Raid-10 any day). You're wise to be doing backups of the thing. Good luck with it. -- -Eric 'shubes' -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
On 10/22/2011 07:12 AM, Andrew Gideon wrote: On Fri, 21 Oct 2011 12:10:09 -0400, Kevin Korb wrote: If you want something you can run after the fact here is a tool I wrote a while back that does a sort of diff across 2 --link-dest based backups: http://sanitarium.net/unix_stuff/rspaghetti_backup/diff_backup.pl.txt It will also tell you what files were not included in the newer backup which --itemize-changes will not since it doesn't actually --delete anything. I like how comprehensive this is, but wouldn't it - by definition of what it means to have a new file in the new directory tree - always report a modified file as having a different inode? That takes me a quick and dirty solution for this. Simply do a find looking for files with a -links (link count) of 1 to see what files are newly copied in the newer directory tree. - Andrew Sorry for re-posting your idea Andrew. Great minds think alike. ;) That sometimes happens, especially in threaded view. -- -Eric 'shubes' -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I would say that not doing backups of RAID at any level is rather unwise. Of course I have seen dual drive failures of a RAID1 array on about 5 occasions. I have also seen a dual drive failure and a controller SNAFU of a RAID5 on 2 occasions. Simply put, RAID is not a backup. RAID is to keep you running when a drive dies. Backups are to get you running again when RAID fails or when something happens that RAID doesn't protect from (file deletion/corruption either accident or on purpose). On 10/22/11 23:03, Eric Shubert wrote: On 10/22/2011 04:04 PM, Ido Magal wrote: 'find' on the droboFS doesn't support '-links' so I had to get creative. And apparently perl has issues on it as well. On Sat, Oct 22, 2011 at 08:02, Eric Shuberte...@shubes.net wrote: # find /backup/dir -type f -links 1 ? Sorry, I'm not familiar with droboFS. So I checked it out a bit. From the looks if things, droboFS may not (from their web site) deliver the best file sharing experience ever. I certainly wouldn't trust my data to BeyondRAID technology alone (give me software Raid-10 any day). You're wise to be doing backups of the thing. Good luck with it. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6jhg4ACgkQVKC1jlbQAQdzxgCfdj3zH10QrDXQtEYJ1iT68mnx XkMAoLckQyItx+lWIK0PskJmcNL8bk1q =V9RQ -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
I agree. Raid is no backup. I expect your experience is extensive to see that many dual drive RAID1 failures. Haven't seen one yet myself (knock wood). Several years ago, I recovered all but a handful of files from a 3-drive raid-5 array where 2 drives had failed. Wasn't pretty. Needless to say, there were no backups (wasn't my doing). I know where you're coming from. ;) -- -Eric 'shubes' On 10/22/2011 08:12 PM, Kevin Korb wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I would say that not doing backups of RAID at any level is rather unwise. Of course I have seen dual drive failures of a RAID1 array on about 5 occasions. I have also seen a dual drive failure and a controller SNAFU of a RAID5 on 2 occasions. Simply put, RAID is not a backup. RAID is to keep you running when a drive dies. Backups are to get you running again when RAID fails or when something happens that RAID doesn't protect from (file deletion/corruption either accident or on purpose). On 10/22/11 23:03, Eric Shubert wrote: On 10/22/2011 04:04 PM, Ido Magal wrote: 'find' on the droboFS doesn't support '-links' so I had to get creative. And apparently perl has issues on it as well. On Sat, Oct 22, 2011 at 08:02, Eric Shuberte...@shubes.net wrote: # find /backup/dir -type f -links 1 ? Sorry, I'm not familiar with droboFS. So I checked it out a bit. From the looks if things, droboFS may not (from their web site) deliver the best file sharing experience ever. I certainly wouldn't trust my data to BeyondRAID technology alone (give me software Raid-10 any day). You're wise to be doing backups of the thing. Good luck with it. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6jhg4ACgkQVKC1jlbQAQdzxgCfdj3zH10QrDXQtEYJ1iT68mnx XkMAoLckQyItx+lWIK0PskJmcNL8bk1q =V9RQ -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
With rsync --link-dest, is it possible to determine which files are new?
Hi, I posted the following question on Stack Overflow and got no response. I'm hoping this list might provide some answers. http://stackoverflow.com/q/7629550/597864 I'm using rsync --link-dest to differentially back up my computer every day. After each backup, I'd like to save out a log of the new/changed files and ignore the linked files. Is this possible? If so, how would I do it? TIA. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The simplest solution is to add --itemize-changes to your backup parameters. That way rsync will give you the list of what files it modified as well as why it needed to modify them (note that if you use - --verbose twice you will see the files that it doesn't touch which isn't what you want). If you want something you can run after the fact here is a tool I wrote a while back that does a sort of diff across 2 --link-dest based backups: http://sanitarium.net/unix_stuff/rspaghetti_backup/diff_backup.pl.txt It will also tell you what files were not included in the newer backup which --itemize-changes will not since it doesn't actually --delete anything. The program is written in perl so it should be easy enough to tweak it if it doesn't do exactly what you want. On 10/21/11 11:11, Ido Magal wrote: Hi, I posted the following question on Stack Overflow and got no response. I'm hoping this list might provide some answers. http://stackoverflow.com/q/7629550/597864 I'm using rsync --link-dest to differentially back up my computer every day. After each backup, I'd like to save out a log of the new/changed files and ignore the linked files. Is this possible? If so, how would I do it? TIA. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6hmWEACgkQVKC1jlbQAQcsJQCeLSn8o7vochObfUKsIbiDeDgd O68Anib1tvAvFxgBvCx16Fs8WsBu1iEf =0Efy -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
Thanks! Unfortunately, it doesn't look like the perl for DroboFS supports the required libraries for your script, error while loading shared libraries: libperl.so.5.10: cannot open shared object file: No such file or directory but --itemize-changes should be sufficient. On Fri, Oct 21, 2011 at 09:10, Kevin Korb k...@sanitarium.net wrote: --itemize-changes -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: With rsync --link-dest, is it possible to determine which files are new?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ummm, I didn't use any perl libraries. The only thing I used that isn't standard perl is that I shell out to the md5sum command rather than using a module for that. On 10/21/11 22:14, Ido Magal wrote: Thanks! Unfortunately, it doesn't look like the perl for DroboFS supports the required libraries for your script, error while loading shared libraries: libperl.so.5.10: cannot open shared object file: No such file or directory but --itemize-changes should be sufficient. On Fri, Oct 21, 2011 at 09:10, Kevin Korb k...@sanitarium.net wrote: --itemize-changes - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6iJ6YACgkQVKC1jlbQAQd6bACgxMCuAagTGWC6IFYXr2u5MD3p 6EAAn20OfmlhtposOtlZENdHvkx9pj6p =xrbY -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
