Update: I investigated bit more on the backgrounds of compression & number of threads, and came to the following conclusion:
Benchmark with *sshfs backend:* Preparing test data... Measuring throughput to cache... Cache throughput with 4 KiB blocks: 12446 KiB/sec Cache throughput with 8 KiB blocks: 15351 KiB/sec Cache throughput with 16 KiB blocks: 18712 KiB/sec Cache throughput with 32 KiB blocks: 20681 KiB/sec Cache throughput with 64 KiB blocks: 20703 KiB/sec Cache throughput with 128 KiB blocks: 20768 KiB/sec Measuring raw backend throughput.. Backend throughput: *8244 KiB/sec* Test file size: 1431.98 MiB compressing with lzma-6... lzma compression speed: 1172 KiB/sec per thread (in) lzma compression speed: 1166 KiB/sec per thread (out) compressing with bzip2-6... bzip2 compression speed: 2076 KiB/sec per thread (in) bzip2 compression speed: 2064 KiB/sec per thread (out) compressing with zlib-6... zlib compression speed: 7921 KiB/sec per thread (in) zlib compression speed: 7884 KiB/sec per thread (out) With 128 KiB blocks, maximum performance for different compression algorithms and thread counts is: Threads: 1 2 4 8 Max FS throughput (lzma): 1172 KiB/s 2344 KiB/s 4688 KiB/s 8287 KiB/s ..limited by: CPU CPU CPU uplink Max FS throughput (bzip2): 2076 KiB/s 4152 KiB/s 8292 KiB/s 8292 KiB/s ..limited by: CPU CPU uplink uplink Max FS throughput (zlib): 7921 KiB/s 8283 KiB/s 8283 KiB/s 8283 KiB/s ..limited by: CPU uplink uplink uplink All numbers assume that the test file is representative and that there are enough processor cores to run all active threads in parallel. To compensate for network latency, you should use about twice as many upload threads as indicated by the above table. Benchmark with *NFS backend:* Preparing test data... Measuring throughput to cache... Cache throughput with 4 KiB blocks: 12409 KiB/sec Cache throughput with 8 KiB blocks: 15658 KiB/sec Cache throughput with 16 KiB blocks: 19004 KiB/sec Cache throughput with 32 KiB blocks: 20691 KiB/sec Cache throughput with 64 KiB blocks: 21500 KiB/sec Cache throughput with 128 KiB blocks: 21915 KiB/sec Measuring raw backend throughput.. Backend throughput: *11120 KiB/sec* Test file size: 1431.98 MiB compressing with lzma-6... lzma compression speed: 1175 KiB/sec per thread (in) lzma compression speed: 1169 KiB/sec per thread (out) compressing with bzip2-6... bzip2 compression speed: 2305 KiB/sec per thread (in) bzip2 compression speed: 2292 KiB/sec per thread (out) compressing with zlib-6... zlib compression speed: 7939 KiB/sec per thread (in) zlib compression speed: 7902 KiB/sec per thread (out) With 128 KiB blocks, maximum performance for different compression algorithms and thread counts is: Threads: 1 2 4 8 Max FS throughput (lzma): 1175 KiB/s 2351 KiB/s 4703 KiB/s 9407 KiB/s ..limited by: CPU CPU CPU CPU Max FS throughput (bzip2): 2305 KiB/s 4611 KiB/s 9222 KiB/s 11185 KiB/s ..limited by: CPU CPU CPU uplink Max FS throughput (zlib): 7939 KiB/s 11172 KiB/s 11172 KiB/s 11172 KiB/s ..limited by: CPU uplink uplink uplink All numbers assume that the test file is representative and that there are enough processor cores to run all active threads in parallel. To compensate for network latency, you should use about twice as many upload threads as indicated by the above table. As far as I can interprete these numbers, the autodetection thread of 4 could be doubled, whereas compression is not of my main concern, I could switch to zlib to tripple the speeds. Is this interpretation correct? Thanks! -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
