translation_file not saved in case of lstat problem
Dear, I' running rsyncrypto with option --name-encrypt=translation_file to encrypt filenames. The file translation_file is in most caes updated at the end of the run. Currently, the translation_file is not updated at the end of the run. There is an error message on 'lstat failed(/dir/dir/filename): No such file or directory' The error is caused by the fact that between rsyncrypto reading the directory and then attempting to access the file, the file has been deleted. So I end up with many encrypted files, but no updated translation_file. How can it be forced that translation_file is written out, even in the case of a lstat error ? regards, Johan ___ Rsyncrypto-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel
Re: translation_file not saved in case of lstat problem
Hello Shachar, as requested, reminder send to the list regards, Johan On 17/09/18 21:32, Shachar Shemesh wrote: Hello Johan, On 16/09/18 17:36, [email protected] wrote: Dear, I' running rsyncrypto with option --name-encrypt=translation_file to encrypt filenames. The file translation_file is in most caes updated at the end of the run. Currently, the translation_file is not updated at the end of the run. There is an error message on 'lstat failed(/dir/dir/filename): No such file or directory' The error is caused by the fact that between rsyncrypto reading the directory and then attempting to access the file, the file has been deleted. So I end up with many encrypted files, but no updated translation_file. How can it be forced that translation_file is written out, even in the case of a lstat error ? regards, Johan It sounds like a bug. I will have to look into it. Sadly, I'm in the middle of an extremely busy time right now, so it will take me a little while. Please, and I do mean it, if you do not hear from me within a week, please please please send me (to the list) a reminder. Thank you for your report, Shachar ___ Rsyncrypto-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel ___ Rsyncrypto-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel
Re: translation_file not saved in case of lstat problem
Hello Shachar, hereby a scenario to reproduce many times. I've created a set-up where you can reproduce numerous time, very quickly. In scenario, I have a huge number of files in a directory, when rsyncrypto is running, I suspend with Ctrl-Z, delete all files in the directory, then continue unsuspending rsyncrypto by bringing to the foreground with 'fg' command. To be able to reproduce time-after-time, I'm keeping a copy of the original big directory in repo-source-dir-copy Set-up of environment in a test-directory: mkdir repo-source-dir mkdir repo-encrypted-dir mkdir SRCDIR.KEYS dd bs=4k count=2 if=/dev/urandom of=./repo-source-dir-copy/orig-file let a=1 cd repo-source-dir-copy while true; do cp orig-file copy-file-$a; let a=a+1; echo $a; done # interrupt when sufficient files created, like 600-700 # cd .. dd bs=4k count=2 if=/dev/urandom of=./repo-source-dir/orig-file2 dd bs=4k count=2 if=/dev/urandom of=./repo-source-dir/orig-file3 ./rsyncrypto -vvr --changed --trim=0 \ --name-encrypt=./repo-encrypt-filename --ne-nesting=3 \ ./repo-source-dir \ ./repo-encrypted-dir \ ./SRCDIR.KEYS \ ./rckey.crt # # now 2 files in repo-encrypt-filename # # strings repo-encrypt-filename # /73B2387C10C76099CDADBC8BE8EB908B ./repo-source-dir/orig-file3 # /B06C02944DC76FCC72B038EEB0FCA02E ./repo-source-dir/orig-file2 = Then to reproduce the problem: # # add many files to repo-source-dir # ln ./repo-source-dir-copy/* ./repo-source-dir # # execute new backup # ./rsyncrypto -vvr --changed --trim=0 \ --name-encrypt=./repo-encrypt-filename --ne-nesting=3 \ ./repo-source-dir \ ./repo-encrypted-dir \ ./SRCDIR.KEYS \ ./rckey.crt # # suspend when running by ctrl-Z # # Encrypting ./repo-source-dir/copy-file-753 # Encrypting ./repo-source-dir/copy-file-629 # ^Z # [1]+ Stopped ./rsyncrypto -vvr --changed --trim=0 --name-encrypt=./repo-encrypt-filename --ne-nesting=3 ./repo-source-dir ./repo-encrypted-dir ./SRCDIR.KEYS ./rckey.crt # rm repo-source-dir/* # # bring back to foreground # # fg # # lstat failed(./repo-source-dir/copy-file-12): No such file or directory # lstat failed(./repo-source-dir/copy-file-178): No such file or directory # lstat failed(./repo-source-dir/copy-file-296): No such file or directory # lstat failed(./repo-source-dir/copy-file-70): No such file or directory # lstat failed(./repo-source-dir/copy-file-456): No such file or directory # lstat failed(./repo-source-dir/copy-file-580): No such file or directory # lstat failed(./repo-source-dir/copy-file-606): No such file or directory # Exit code delayed from previous errors # # check number of filename in repo-encrypt-filename # # strings repo-encrypt-filename # /73B2387C10C76099CDADBC8BE8EB908B ./repo-source-dir/orig-file3 # /B06C02944DC76FCC72B038EEB0FCA02E ./repo-source-dir/orig-file2 # == If you need to test again, start over at 'ln' command regards, Johan On 05/10/18 20:48, Shachar Shemesh wrote: On 16/09/18 17:36, [email protected] wrote: Dear, I' running rsyncrypto with option --name-encrypt=translation_file to encrypt filenames. The file translation_file is in most caes updated at the end of the run. Currently, the translation_file is not updated at the end of the run. There is an error message on 'lstat failed(/dir/dir/filename): No such file or directory' The error is caused by the fact that between rsyncrypto reading the directory and then attempting to access the file, the file has been deleted. So I end up with many encrypted files, but no updated translation_file. How can it be forced that translation_file is written out, even in the case of a lstat error ? regards, Johan Hello Johan, I'm trying to wrap my brains about the sequence of events. Can you give me reproduction instructions to this problem? What steps do I need to take in order to see it on my system? Thanks, Shachar ___ Rsyncrypto-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel ___ Rsyncrypto-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel
Re: translation_file not saved in case of lstat problem
On 07/10/18 09:40, Shachar Shemesh wrote: On 06/10/18 11:51, [email protected] wrote: Hello Shachar, hereby a scenario to reproduce many times. I've created a set-up where you can reproduce numerous time, very quickly. In scenario, I have a huge number of files in a directory, when rsyncrypto is running, I suspend with Ctrl-Z, delete all files in the directory, then continue unsuspending rsyncrypto by bringing to the foreground with 'fg' command. To be able to reproduce time-after-time, I'm keeping a copy of the original big directory in repo-source-dir-copy Hello Johan, I understand the scenario you describe. I just don't understand how rsyncrypto is supposed to survive it. As with many other programs, rsyncrypto expects the files it needs not to shift around /while/ it is manipulating them. Essentially, you are asking that if you delete the file rsyncrypto is creating, that it not get deleted. Am I missing something? Can you explain the real wold aim you are trying to achieve that causes this condition to trigger? Thank you, Shachar Hello Shachar, if during a run, there are hunderds of additional files encrypted and sync, and there is one file that went missing causing the lstat error, rsyncrypto behaves this lstat error as a fatal error and doesn't update the translation file. In addition, these hundreds of additional files are now encrypted and can't be matched back to the original filename. Also at the next run, rsyncrypto will create an additional copy of these extra files. What I'm asking for is that rsyncrypto doesn't handle this as a fatal error and at the end of the run still update the translation file when the lstat error occured. I discovered the issue when running rsyncrypto on my home directory when still working on my box which causes sometimes that a file gets deleted. The problem also happens when using --filelist, if there is an error in that filelist, and there is a file that does not exist, no translation file is saved, here output of such: ./rsyncrypto -vv --changed --trim=0 --name-encrypt=./repo-encrypt-filename --ne-nesting=3 --filelist=file-list . ./repo-encrypted-dir ./SRCDIR.KEYS ./rckey.crt Encrypting file: ./repo-src-dir2/orig-file3 Encrypting file: ./repo-src-dir2/orig-file4 Error in encryption of ./repo-src-dir2/orig-file5: stat failed(././repo-src-dir2/orig-file5): No such file or directory Exit code delayed from previous errors A fix that I've found is to change the end of main.cpp to (and move declaration of rsa_key to a higher level): } catch( const rscerror &err ) { std::cerr< file_encrypt(FILENAME(nameenc), autofd::combine_paths(FILENAME(dst), FILEMAPNAME). c_str(), autofd::combine_paths(FILENAME(key), FILEMAPNAME).c_str(), rsa_key, NULL ); } } return ret; } Probably some extra logic to be added to only write translation file in case of lstat error, not in case of other errors. regards, Johan ___ Rsyncrypto-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel ___ Rsyncrypto-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel
Re: translation_file not saved in case of lstat problem
On 08/10/18 05:54, Shachar Shemesh wrote: I want to clarify the report. You are not complaining about someone erasing the name translation file during encryption. You are complaining about someone erasing one of the *source* files. Is that correct? correct, this is about a *source* file that is erased while rsyncrypto is running. The other situation where there is a problem, is when the *filelist* contains a file which does not exist (which could have been caused by someone erasing the *source* file, or the filelist was badly created). Erasing the name translation file or one of the encrypted files during encryption is not a bug rsyncrypto is supposed to handle. No one should be touching those during the encryption phase. Agree, that scenario is not a rsyncrypto bug, that is user error/bug. Erasing a source file is something that rsyncrypto should, absolutely, be able to handle without failing outright. Also I fully agree here, rsyncrypto should be able to handle this. Thank you, Shachar thank you for working and supporting this Johan On 07/10/18 11:14, [email protected] wrote: On 07/10/18 09:40, Shachar Shemesh wrote: On 06/10/18 11:51, [email protected] wrote: Hello Shachar, hereby a scenario to reproduce many times. I've created a set-up where you can reproduce numerous time, very quickly. In scenario, I have a huge number of files in a directory, when rsyncrypto is running, I suspend with Ctrl-Z, delete all files in the directory, then continue unsuspending rsyncrypto by bringing to the foreground with 'fg' command. To be able to reproduce time-after-time, I'm keeping a copy of the original big directory in repo-source-dir-copy Hello Johan, I understand the scenario you describe. I just don't understand how rsyncrypto is supposed to survive it. As with many other programs, rsyncrypto expects the files it needs not to shift around /while/ it is manipulating them. Essentially, you are asking that if you delete the file rsyncrypto is creating, that it not get deleted. Am I missing something? Can you explain the real wold aim you are trying to achieve that causes this condition to trigger? Thank you, Shachar Hello Shachar, if during a run, there are hunderds of additional files encrypted and sync, and there is one file that went missing causing the lstat error, rsyncrypto behaves this lstat error as a fatal error and doesn't update the translation file. In addition, these hundreds of additional files are now encrypted and can't be matched back to the original filename. Also at the next run, rsyncrypto will create an additional copy of these extra files. What I'm asking for is that rsyncrypto doesn't handle this as a fatal error and at the end of the run still update the translation file when the lstat error occured. I discovered the issue when running rsyncrypto on my home directory when still working on my box which causes sometimes that a file gets deleted. The problem also happens when using --filelist, if there is an error in that filelist, and there is a file that does not exist, no translation file is saved, here output of such: ./rsyncrypto -vv --changed --trim=0 --name-encrypt=./repo-encrypt-filename --ne-nesting=3 --filelist=file-list . ./repo-encrypted-dir ./SRCDIR.KEYS ./rckey.crt Encrypting file: ./repo-src-dir2/orig-file3 Encrypting file: ./repo-src-dir2/orig-file4 Error in encryption of ./repo-src-dir2/orig-file5: stat failed(././repo-src-dir2/orig-file5): No such file or directory Exit code delayed from previous errors A fix that I've found is to change the end of main.cpp to (and move declaration of rsa_key to a higher level): } catch( const rscerror &err ) { std::cerr< file_encrypt(FILENAME(nameenc), autofd::combine_paths(FILENAME(dst), FILEMAPNAME). c_str(), autofd::combine_paths(FILENAME(key), FILEMAPNAME).c_str(), rsa_key, NULL ); } } return ret; } Probably some extra logic to be added to only write translation file in case of lstat error, not in case of other errors. regards, Johan ___ Rsyncrypto-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel ___ Rsyncrypto-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel ___ Rsyncrypto-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel ___ Rsyncrypto-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel
