We have our local swift cluster and we have a few use cases for s3ql. One of them is keeping a mirror of a posix file system. For this we want to maximize write throughput per process. I am aware that each copy process is limited by the write throughput of the swift cluster which is between 50 and 60 MB/s. I thought that compression would be able to improve upload performance but no matter which compression type I use my 1GB test file always needs 24-28 sec to upload and rsync shows between 30 and 40 MB/s throughput. Is zlib perhaps too slow to do real on-the-fly compression and has to do most of the work after the file is uploaded?
root@box:/loc/tmp# s3qlstat --debug /mnt/Swift/s3ql_zlib_0 Directory entries: 2 Inodes: 4 Data blocks: 8 Total data size: 953 MiB After de-duplication: 953 MiB (100.00% of total) After compression: 953 MiB (100.02% of total, 100.02% of de-duplicated) Database size: 64 KiB (uncompressed) Cache usage: 953 MiB (dirty: 0 bytes) root@box:/loc/tmp# s3qlstat --debug /mnt/Swift/s3ql_zlib_1/ Directory entries: 2 Inodes: 4 Data blocks: 8 Total data size: 953 MiB After de-duplication: 953 MiB (100.00% of total) After compression: 316 MiB (33.24% of total, 33.24% of de-duplicated) Database size: 64 KiB (uncompressed) Cache usage: 953 MiB (dirty: 0 bytes) root@box:/loc/tmp# s3qlstat --debug /mnt/Swift/s3ql_zlib_6 Directory entries: 2 Inodes: 4 Data blocks: 8 Total data size: 953 MiB After de-duplication: 953 MiB (100.00% of total) After compression: 278 MiB (29.24% of total, 29.24% of de-duplicated) Database size: 64 KiB (uncompressed) Cache usage: 953 MiB (dirty: 0 bytes) root@box:/loc/tmp# s3qlstat --debug /mnt/Swift/s3ql_bzip2_9/ Directory entries: 2 Inodes: 4 Data blocks: 8 Total data size: 953 MiB After de-duplication: 953 MiB (100.00% of total) After compression: 260 MiB (27.34% of total, 27.34% of de-duplicated) Database size: 64 KiB (uncompressed) Cache usage: 953 MiB (dirty: 0 bytes) -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
