I am trying to rsync between s3ql on a local disk to another s3ql on google
storage
The rsync is SUPER slow with pushing 200GB over a week or so.....
Any tuning can we do with it?
Thanks!
Here is how the s3ql setting:
Local:
exec mount.s3ql --upstart --cachedir "$cache" --threads 16 --compress
zlib --log syslog "$backup_url" "$backup_dir"
Google:
exec mount.s3ql --upstart --cachedir "$cache" --threads 16 --log syslog
"$offsite_url" "$offsite_dir"
s3qlstat on local disk:
Preparing test data...
Measuring throughput to cache...
Cache throughput with 4 KiB blocks: 11885 KiB/sec
Cache throughput with 8 KiB blocks: 25585 KiB/sec
Cache throughput with 16 KiB blocks: 56202 KiB/sec
Cache throughput with 32 KiB blocks: 87566 KiB/sec
Cache throughput with 64 KiB blocks: 135954 KiB/sec
Cache throughput with 128 KiB blocks: 184896 KiB/sec
Measuring raw backend throughput..
Backend throughput: 1105161 KiB/sec
Test file size: 0.00 MiB
compressing with lzma-6...
lzma compression speed: 1310 KiB/sec per thread (in)
lzma compression speed: 436 KiB/sec per thread (out)
compressing with bzip2-6...
bzip2 compression speed: 1163 KiB/sec per thread (in)
bzip2 compression speed: 311 KiB/sec per thread (out)
compressing with zlib-6...
zlib compression speed: 3737 KiB/sec per thread (in)
zlib compression speed: 745 KiB/sec per thread (out)
With 128 KiB blocks, maximum performance for different compression
algorithms and thread counts is:
Threads: 1 2 4 8
16
Max FS throughput (lzma): 1310 KiB/s 2620 KiB/s 5240 KiB/s 10481
KiB/s 20963 KiB/s
..limited by: CPU CPU CPU CPU
CPU
Max FS throughput (bzip2): 1163 KiB/s 2327 KiB/s 4654 KiB/s 9308
KiB/s 18617 KiB/s
..limited by: CPU CPU CPU CPU
CPU
Max FS throughput (zlib): 3737 KiB/s 7474 KiB/s 14949 KiB/s 29898
KiB/s 59797 KiB/s
..limited by: CPU CPU CPU CPU
CPU
s3qlstat on google storage:
Preparing test data...
Measuring throughput to cache...
Cache throughput with 4 KiB blocks: 12198 KiB/sec
Cache throughput with 8 KiB blocks: 24942 KiB/sec
Cache throughput with 16 KiB blocks: 49939 KiB/sec
Cache throughput with 32 KiB blocks: 85977 KiB/sec
Cache throughput with 64 KiB blocks: 132026 KiB/sec
Cache throughput with 128 KiB blocks: 187159 KiB/sec
Measuring raw backend throughput..
Backend throughput: 41466 KiB/sec
Test file size: 0.00 MiB
compressing with lzma-6...
lzma compression speed: 654 KiB/sec per thread (in)
lzma compression speed: 218 KiB/sec per thread (out)
compressing with bzip2-6...
bzip2 compression speed: 757 KiB/sec per thread (in)
bzip2 compression speed: 202 KiB/sec per thread (out)
compressing with zlib-6...
zlib compression speed: 1924 KiB/sec per thread (in)
zlib compression speed: 383 KiB/sec per thread (out)
With 128 KiB blocks, maximum performance for different compression
algorithms and thread counts is:
Threads: 1 2 4 8
16
Max FS throughput (lzma): 654 KiB/s 1309 KiB/s 2618 KiB/s 5236
KiB/s 10473 KiB/s
..limited by: CPU CPU CPU CPU
CPU
Max FS throughput (bzip2): 757 KiB/s 1514 KiB/s 3028 KiB/s 6057
KiB/s 12115 KiB/s
..limited by: CPU CPU CPU CPU
CPU
Max FS throughput (zlib): 1924 KiB/s 3848 KiB/s 7696 KiB/s 15392
KiB/s 30785 KiB/s
..limited by: CPU CPU CPU CPU
CPU
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
