This is the rsync arguments we using /usr/bin/rsync -aHAXx --numeric-ids --delete
and it's running 10 threads at a time Thanks -Peter On Monday, June 20, 2016 at 11:52:40 AM UTC-7, Peter Auyeung wrote: > > I am trying to rsync between s3ql on a local disk to another s3ql on > google storage > The rsync is SUPER slow with pushing 200GB over a week or so..... > > Any tuning can we do with it? > > Thanks! > > Here is how the s3ql setting: > > Local: > exec mount.s3ql --upstart --cachedir "$cache" --threads 16 --compress > zlib --log syslog "$backup_url" "$backup_dir" > > Google: > exec mount.s3ql --upstart --cachedir "$cache" --threads 16 --log > syslog "$offsite_url" "$offsite_dir" > > > s3qlstat on local disk: > Preparing test data... > Measuring throughput to cache... > Cache throughput with 4 KiB blocks: 11885 KiB/sec > Cache throughput with 8 KiB blocks: 25585 KiB/sec > Cache throughput with 16 KiB blocks: 56202 KiB/sec > Cache throughput with 32 KiB blocks: 87566 KiB/sec > Cache throughput with 64 KiB blocks: 135954 KiB/sec > Cache throughput with 128 KiB blocks: 184896 KiB/sec > Measuring raw backend throughput.. > Backend throughput: 1105161 KiB/sec > Test file size: 0.00 MiB > compressing with lzma-6... > lzma compression speed: 1310 KiB/sec per thread (in) > lzma compression speed: 436 KiB/sec per thread (out) > compressing with bzip2-6... > bzip2 compression speed: 1163 KiB/sec per thread (in) > bzip2 compression speed: 311 KiB/sec per thread (out) > compressing with zlib-6... > zlib compression speed: 3737 KiB/sec per thread (in) > zlib compression speed: 745 KiB/sec per thread (out) > > With 128 KiB blocks, maximum performance for different compression > algorithms and thread counts is: > > Threads: 1 2 4 > 8 16 > Max FS throughput (lzma): 1310 KiB/s 2620 KiB/s 5240 KiB/s 10481 > KiB/s 20963 KiB/s > ..limited by: CPU CPU CPU > CPU CPU > Max FS throughput (bzip2): 1163 KiB/s 2327 KiB/s 4654 KiB/s 9308 > KiB/s 18617 KiB/s > ..limited by: CPU CPU CPU > CPU CPU > Max FS throughput (zlib): 3737 KiB/s 7474 KiB/s 14949 KiB/s 29898 > KiB/s 59797 KiB/s > ..limited by: CPU CPU CPU > CPU CPU > > > s3qlstat on google storage: > Preparing test data... > Measuring throughput to cache... > Cache throughput with 4 KiB blocks: 12198 KiB/sec > Cache throughput with 8 KiB blocks: 24942 KiB/sec > Cache throughput with 16 KiB blocks: 49939 KiB/sec > Cache throughput with 32 KiB blocks: 85977 KiB/sec > Cache throughput with 64 KiB blocks: 132026 KiB/sec > Cache throughput with 128 KiB blocks: 187159 KiB/sec > Measuring raw backend throughput.. > Backend throughput: 41466 KiB/sec > Test file size: 0.00 MiB > compressing with lzma-6... > lzma compression speed: 654 KiB/sec per thread (in) > lzma compression speed: 218 KiB/sec per thread (out) > compressing with bzip2-6... > bzip2 compression speed: 757 KiB/sec per thread (in) > bzip2 compression speed: 202 KiB/sec per thread (out) > compressing with zlib-6... > zlib compression speed: 1924 KiB/sec per thread (in) > zlib compression speed: 383 KiB/sec per thread (out) > > With 128 KiB blocks, maximum performance for different compression > algorithms and thread counts is: > > Threads: 1 2 4 > 8 16 > Max FS throughput (lzma): 654 KiB/s 1309 KiB/s 2618 KiB/s 5236 > KiB/s 10473 KiB/s > ..limited by: CPU CPU CPU > CPU CPU > Max FS throughput (bzip2): 757 KiB/s 1514 KiB/s 3028 KiB/s 6057 > KiB/s 12115 KiB/s > ..limited by: CPU CPU CPU > CPU CPU > Max FS throughput (zlib): 1924 KiB/s 3848 KiB/s 7696 KiB/s 15392 > KiB/s 30785 KiB/s > ..limited by: CPU CPU CPU > CPU CPU > > -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
