David Zeillinger wrote:
Did you happen to investigate why rsync -S is taking so much time? If it doesn't deal with sparse file the way one expects, this option is probably broken. Also have you already tried something like the advice in http://lists.samba.org/archive/rsync/2003-August/007000.html ?
It takes a long time because it needs to process the full files anyway. You can test that if you want to see it.
Anyway, I think the way to go is using tar. It preserves the sparseness property of the files, so something like this could work: If you tar the file without using compression, you would get a file the size of the sparsefile, with a lot of zeroes in it. Then use a run-length encoding on it to collapse the zeroes.
That goes against the first goal of the question to find the sparse files for example. And tar doesn't remove the use of resource to tar it anyway on either side and rsync also can use compression on the fly as well witch it is in use as well already. And I am not sure it would work anyway as if you think about it for a minute really. What's the difference to copy a sparse file via scp, rsync or untar it. Why would scp and rsync runs out of space to start with if they are copying empty space, or pointer to empty block and jam in the process???
If you think about that, why would tar does a better job? No I didn't try it and may be I will just to know, not as a solutions however. The solution is to not copy the sparse file in the first place when possible and that's what I am working on.
The interesting question for me that still pending and that I do not have answer for is more this.
If sparse file is pointers only to empty drive space not use, then why would scp run out of space copying empty pointers in the first place?
That's really the interesting question this brings to me in the process here.
There was more, but I got my answer to them so far.
Sync this file with rsync.
in my opinion, you only make the problem worst, however, I will need to test it to talk knowingly. rsync already compress it at the source and I proof in the various tests I sent to the list that it doesn't send more data across the link when -S is use, but only when it wasn't. So, tar it before wouldn't change that and this would only add more step to the process.
Daniel
