Re: [Duplicity-team] [Question #163116]: Invalid data - SHA1 hash mismatch when trying to verify/restore
Question #163116 on Duplicity changed: https://answers.launchpad.net/duplicity/+question/163116 edso posted a new comment: a patch that makes the mismatch output more verbose is here https://code.launchpad.net/~ed.so/duplicity/numowner+hashverbose output is now: Invalid data - SHA1 hash mismatch for file: duplicity-full.20110818T165723Z.vol1.difftar.gz Calculated hash: b0f110be2e2d08d8bf35cc9b736d4ac252a31922 Manifest hash: 00f0e0b93552a289da8b41104698df5b78f9ba65 hope this helps future victims .. ede/duply.net -- You received this question notification because you are a member of duplicity-team, which is an answer contact for Duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Question #163116]: Invalid data - SHA1 hash mismatch when trying to verify/restore
Question #163116 on Duplicity changed: https://answers.launchpad.net/duplicity/+question/163116 edso posted a new comment: Eventually I ran a verification to a date prior to the corrupted manifest, the one with the missmatched hash, and voila! Verification completed and then a restore to that date was successful. I will assume good to hear and makes perfect sense. Although I was lucky this time and I only lost some history (and a lot of time), duplicity failed to 1) detect there was a problem during the normal backup cycle and notify it there was no obvious problem during backing up, but during restoring, which is no part of backing up and 2) provide a clear indication of why verification was failing and which files/volumes where being affected-event at -v9. you're right the hash mismatch should be more talkative. I'm now updating my backup script to run frequent verifications of my backends. This is something I can afford because I run a local file server. make the chains as short as you can afford and you are as safe as you can get Overall my gratitude goes out to Ken and Edso for helping through this process and suggesting valuable pointers. pleasure, ede/duply.net -- You received this question notification because you are a member of duplicity-team, which is an answer contact for Duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Question #163116]: Invalid data - SHA1 hash mismatch when trying to verify/restore
Question #163116 on Duplicity changed: https://answers.launchpad.net/duplicity/+question/163116 Status: Answered = Open Rodrigo Alvarez is still having a problem: Hi Ken, I looked around, and updated by adding: root hard nofile 5000 * hard nofile 5000 to /etc/security/limits.conf as well as by running ulimit -n 5 inside /root/.bashrc After rebooting I get 5 when I run ulimit -n for root or any other user yet my attempt to verify/restore my backup set still fails with the OS 24 error. Could my backup be corrupted? Other similar backup sets do verify without problems. Could there be a bug lurking around? Thank you very much for your help so far. Your responsiveness is duplicity's greatest feature. -- You received this question notification because you are a member of duplicity-team, which is an answer contact for Duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Question #163116]: Invalid data - SHA1 hash mismatch when trying to verify/restore
Question #163116 on Duplicity changed: https://answers.launchpad.net/duplicity/+question/163116 Status: Answered = Open Rodrigo Alvarez is still having a problem: Patching the manifest worked and I was able to move past Vol2. However, now I'm getting an OSError: [Errno 24] Too many open files. I have increased my user's limit for open files from 1024 to 5 (fifty thousand) and this does not change anything. During the process, duplicity uses about 30 GB of space on the scratch drive (75GB are still available). Please help: the complete output log can be found here: http://www.stanford.edu/~rodrigoa/Temp/duplicity_verify_log_2011-07-16.txt For posterity: To patch the manifest I first had to find the corrupted manifest. I did this by copying the gpged manifests, decrypting them, concatenating them and searching for the wrong Hash. Once I identified the suspect manuscript I patched it, encrypted it and replaced the original. Finally I replaced the patched manifest in duplicity's cache so that it would be picked up during the process. $ temp_location=temp_manifests2/ $ full_file=all_manifests $ # Copy Original Manifest to temp location $ cp *.manifest.gpg $temp_location $ cd $temp_location $ gpg --multifile --decrypt *.gpg $ rm *.gpg $ i=`ls`; for f in $i; do echo -e \n\n$f $full_file; cat $f $full_file;done $ # Used gedit to find the wrong hash in all manuscripts and then patched the suspect manuscript file $ gpg --symmetric Manifest_file_name $ # Finaly delete duplicity cache so that it uses the patched manifest. -- You received this question notification because you are a member of duplicity-team, which is an answer contact for Duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Question #163116]: Invalid data - SHA1 hash mismatch when trying to verify/restore
How did you increase ulimit? Did you sign out and back in? Increasing files is a very cheap setting resource wise. You can find out how to increase it permanently by searching Google for 'linux num_files limit'. Given the number of incrementals you have, duplicity will use a couple of thousand file handles, 3 or 4 per incremental, so 50k is way over, ...Ken On Sat, Jul 16, 2011 at 9:01 AM, Rodrigo Alvarez question163...@answers.launchpad.net wrote: Question #163116 on Duplicity changed: https://answers.launchpad.net/duplicity/+question/163116 Status: Answered = Open Rodrigo Alvarez is still having a problem: Patching the manifest worked and I was able to move past Vol2. However, now I'm getting an OSError: [Errno 24] Too many open files. I have increased my user's limit for open files from 1024 to 5 (fifty thousand) and this does not change anything. During the process, duplicity uses about 30 GB of space on the scratch drive (75GB are still available). Please help: the complete output log can be found here: http://www.stanford.edu/~rodrigoa/Temp/duplicity_verify_log_2011-07-16.txt For posterity: To patch the manifest I first had to find the corrupted manifest. I did this by copying the gpged manifests, decrypting them, concatenating them and searching for the wrong Hash. Once I identified the suspect manuscript I patched it, encrypted it and replaced the original. Finally I replaced the patched manifest in duplicity's cache so that it would be picked up during the process. $ temp_location=temp_manifests2/ $ full_file=all_manifests $ # Copy Original Manifest to temp location $ cp *.manifest.gpg $temp_location $ cd $temp_location $ gpg --multifile --decrypt *.gpg $ rm *.gpg $ i=`ls`; for f in $i; do echo -e \n\n$f $full_file; cat $f $full_file;done $ # Used gedit to find the wrong hash in all manuscripts and then patched the suspect manuscript file $ gpg --symmetric Manifest_file_name $ # Finaly delete duplicity cache so that it uses the patched manifest. -- You received this question notification because you are a member of duplicity-team, which is an answer contact for Duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Question #163116]: Invalid data - SHA1 hash mismatch when trying to verify/restore
It should be 'gpg -e -r key __.manifest' where key is the same key you used for the backup. ...Ken On Sun, Jul 10, 2011 at 12:31 PM, Rodrigo Alvarez question163...@answers.launchpad.net wrote: Question #163116 on Duplicity changed: https://answers.launchpad.net/duplicity/+question/163116 Status: Answered = Open Rodrigo Alvarez is still having a problem: Hi, the manifest patching is going ok. I've found the suspect manifest, decrypted it and patched it but I can't figure out how to encrypt it again so that duplicity picks up the patched manifest. gpg is asking of a user: What gpg command should I run to encrypt the patched manifest? -- You received this question notification because you are a member of duplicity-team, which is an answer contact for Duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Question #163116]: Invalid data - SHA1 hash mismatch when trying to verify/restore
Question #163116 on Duplicity changed: https://answers.launchpad.net/duplicity/+question/163116 Status: Open = Answered Kenneth Loafman proposed the following answer: It should be 'gpg -e -r key __.manifest' where key is the same key you used for the backup. ...Ken On Sun, Jul 10, 2011 at 12:31 PM, Rodrigo Alvarez question163...@answers.launchpad.net wrote: Question #163116 on Duplicity changed: https://answers.launchpad.net/duplicity/+question/163116 Status: Answered = Open Rodrigo Alvarez is still having a problem: Hi, the manifest patching is going ok. I've found the suspect manifest, decrypted it and patched it but I can't figure out how to encrypt it again so that duplicity picks up the patched manifest. gpg is asking of a user: What gpg command should I run to encrypt the patched manifest? -- You received this question notification because you are a member of duplicity-team, which is an answer contact for Duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp -- You received this question notification because you are a member of duplicity-team, which is an answer contact for Duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
Re: [Duplicity-team] [Question #163116]: Invalid data - SHA1 hash mismatch when trying to verify/restore
Question #163116 on Duplicity changed: https://answers.launchpad.net/duplicity/+question/163116 Rodrigo Alvarez posted a new comment: Thanks Kenneth and Edso. Over the weekend I'll try your suggestions and will post the result of the test and Kenneth's request. Thanks again! -- You received this question notification because you are a member of duplicity-team, which is an answer contact for Duplicity. ___ Mailing list: https://launchpad.net/~duplicity-team Post to : duplicity-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~duplicity-team More help : https://help.launchpad.net/ListHelp
