Hi Nikolaus,
[...] is there a reason to not just bump the hardcoded buffer size?
Adding an option typically means that 50% of the people go with the
default (even when it's sub-optimal), 40% pick a value that's even
worse, and 10% actually benefit. So I'd like to avoid options whenever
possible.[...]
If changing the hard-coded BUFSIZE is an option, that's what I would
prefer, too. That would increase the required memory for S3QL, tho. So
when low-memory systems are a of any concern, increasing the BUFSIZE
would be bad for them.
In this use case I use S3QL as a target for Bareos backups. Each backup
is one single file that can get hundreds of GB big. Thus I chose a
max-obj-size of 3GB, no compression (Bareos does that already) and a
cache size of 100 GB. The file systems look like these (cache gets
dropped between backup windows):
Directory entries: 4735
Inodes: 4737
Data blocks: 6211
Total data size: 6.35 TB
After de-duplication: 6.35 TB (100.00% of total)
After compression: 6.35 TB (99.94% of total, 99.94% of de-duplicated)
Database size: 1.91 MiB (uncompressed)
Cache size: 0 bytes, 0 entries
Cache size (dirty): 0 bytes, 0 entries
Queued object removals: 0
So there are relatively few objects/data blocks but they are 1GB on
average. This is quite a different use case as than the default
max-obj-size of 10MB.
Before bumping the BUFSIZE we definitely should benchmark with the
default max-obj-size, too.
Looking at contrib/benchmark.py, I can probably change this to also
benchmark different BUFSIZEs for the upload so that we can get some data
from different configurations.
References:
https://github.com/python/cpython/commit/4f1903061877776973c1bbfadd3d3f146920856e
increased buffer from 16KB to 64KB
https://blogs.blumetech.com/blumetechs-tech-blog/2011/05/faster-python-file-copy.html
10 MB buffer for large files
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/s3ql/3eb1cf7f-de64-639a-09b6-23333d06ec63%40jagszent.de.
For more options, visit https://groups.google.com/d/optout.
