On Mon, 11 Apr 2011, Stephen John Smoogen wrote:
On Mon, Apr 11, 2011 at 05:08, Federico Alves <[email protected]> wrote:
The limitation is not my hardware. The servers are both Dell R900 with SAS
disk arrays. Also, from a Windows virtual machine, inside the same server,
I get around 400 MB speed using FTP transfer, windows to windows. There
must be a different way to do this from Linux.The files are sparse files,
and I need to keep them that way, that's why I use rsync.
Well transferring sparse files is going to be slow and it could be
hardware (unless you are somehow testing with windows of copying
sparse files over). rsync is having to see what real bits are there
and what is fluff so it is going to be CPU and disk intensive.
Just to add to that the rsync --sparse option makes the *transfer* of
sparse data efficient, but unless it has an efficient way to detect the
unallocted areas it will still need to read through the file looking for
blank bits.
Even having to do that the --sparse will save effort since at least it
won't need to *transfer* the large blocks of emptyness to the far end...
Newer versions of rsync have an option to specify the sparse-blocksize to
use, which apparently can vastly improve things on some high-end hardware.
At least some file-systems allow the detection of whch blocks are actually
allocated and in theory rsync could use those but it wouldn't be portable.
e.g. for Solaris' zfs Sun added SEEK_HOLE and SEEK_DATA to their lseek
options which is probably better than the various other hacks that have
been implemented over the years but I don't think it is widely implemented
outside of Solaris... e.g. see:
http://lwn.net/Articles/260795/
which has a description of SEEK_HOLE/DATA and the FIEMAP ioctl() which can
already be used on some (I don't know which) linux fs... I doubt that sl5
will be likely to support either, and I wouldn't like to guess about sl6.
-- Jon
