Julian,
Both scenario's you describe are the way rsync should behave, but it does
not. Let me just describe how case a can be reproduced.
Take a large zip file, my file is now 100MB, and encrypt it with rsyncrypto.
Now rename that file to 1.zip. Encrypt this file again and rename the
encrypted file to 2.zip. Both files should be the same (except the first
blocks), right ?
Copy file 1.zip to test.zip and copy it over to another box with a rsync
command similar to this one.
rsync -rlt -v -P -e ssh Test.zip <user>@<IP>:/data
Notice that you get full performance during the transfer, in my case this is
something like 1.66MB/s. Interrupt the transfer when it is half way down.
Start the transfer again with this command, I added the bwlimit to slow down
the transfer.
rsync -rlt -v -P --bwlimit=12 -e ssh Test.zip <user>@<IP>:/data
The beginning will be fast, like 20MB/s on my infrastructure. Rsync is now
reusing the partially transferred file, which is still on the box. When the
new tmp file has the same size as the partially transferred file, rsync will
start copying over data, conform the bwlimit parameter. So far so good, this
is correct rsync behavior.
Now I will describe the error situation with rsyncrypto encrypted files.
Copy file 1.zip to test.zip and copy it over to another box with a rsync
command similar to this one.
rsync -rlt -v -P -e ssh Test.zip <user>@<IP>:/data
Notice that you get full performance during the transfer, in my case this is
again something like 1.66MB/s. Interrupt the transfer when it is half way
down. Now remove Test.zip and copy 2.zip to Test.zip (file should be the
same). Start the transfer of Test.zip with rsync with this command.
rsync -rlt -v -P --bwlimit=12 -e ssh Test.zip <user>@<IP>:/data
Watch the progress, it is 12kB/s all the way. Rsync is not using the
partially transferred file, it starts at the beginning with a speed
indicated by the bwlimit parameter. I would have expected that in the second
situation the behavior is the same as in the first situation.
Hope it is a bit more clear now.
Regards,
Jan
_____
Van: Julian Pace Ross [mailto:julian.pacer...@gmail.com]
Verzonden: vrijdag 31 juli 2009 9:34
Aan: Jan Alphenaar
CC: L-rsyncrypto
Onderwerp: Re: Partial transfer of rsyncrypto encrypted files
Case A:
Rsync still has to start from the beginning, but it uses the partial file as
a basis file, so transfer of the first part should be much faster than
transferring from scratch. Rsync will NOT do what you are thinking and use
the same file and just continue building on it, but it will start a new
file, leaving the "partial" file there until the new one is completely
rebuilt, after which the partial one is deleted.
Case B:
Same issue really here... Re-encrypting will generate a new file with new
mod timestamp. Again it will be uploaded using already present file as a
basis, making the transfer much faster, but the whole file will be rebuilt
on the destination (of course be using large chunks of the basis file, which
is what will speed it up).
On Thu, Jul 30, 2009 at 11:33 PM, Jan Alphenaar
<jan.alphen...@dotcolour.com> wrote:
Hi,
I was recently playing around with rsyncrypto and rsync and found some
interesting behavior during one of my large file transfers.
This is the situation. After encrypting my large file (approximately 1GB),
rsync starts copying this file to my server. Here is the rsync command used,
I modified it a bit to make it easier to read. Note, the "partial" option is
used. In case the transfer is interrupted, rsync leaves the partially
transferred file on the server.
rsync -rlt -v --partial -e ssh Test.zip jan@<servername>:/data
Half way down I interrupt the transfer, the partially transferred file stays
on the server as expected. In the next session the file is again encrypted
with rsyncrypto. Then I start a new rsync copy of the file, expecting rsync
would continue with the partially transferred file, but it does not, it
starts over at the beginning. Lets call this case a.
It becomes even more fuzzy. Consider the following case (b). When I encrypt
a large file with rsyncrypto and fully transfer it to my server. After
transferring, I am going to encrypt my file again. Start a new transfer of
the newly encrypted file. Now rsync is not copying over the file, fully
making use of the rsync algorithm.
I know it looks like an rsync issue, but the rsync behavior is correct if
the file is not encrypted a second time. Then is does not start at the
beginning (but at the end) the second time it tries to copy a large file.
Can anybody explain this behaviour, cause I am totally in the dark with this
behavior.
Any help is greatly appreciated.
Warm regards,
Jan
----------------------------------------------------------------------------
--
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus
on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Rsyncrypto-devel mailing list
Rsyncrypto-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Rsyncrypto-devel mailing list
Rsyncrypto-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel
