On 2020-07-08 at 12:43 +0300, Cliff Stanford wrote: > On 08/07/2020 12:02, Ivan Shapovalov wrote: > > > Is it correct that s3ql will always download the whole block into > > the > > local cache upon a read that intersects with this block? > > A block is either a single file or part of a file, if the file > exceeds > the maximum block size. There are never multiple files in a block > and > blocks are variable size, up to the maximum.
That's clear enough. What I'm asking is that, for example, if I have 100MiB block size set at mkfs time and 1GiB file on the filesystem (which is thus broken into 10 blocks, let's ignore dedup) and I'm read()ing 10 MiB from the middle of the file, how much data will s3ql download? 10 MiB or 100-200 MiB? > > > If true, then how scalable is s3ql with respect to number of blocks > > in > > the filesystem? That is, how far can I realistically reduce the > > block > > size if my dataset is, say, 10-20 TB? > > > > Basically, I'm trying to optimize for random reads. > > I don't think that reducing the maximum block size will improve > things > for you. If what I conjectured above is true, then reducing block size to average I/O size will obviously reduce download overhead, no? -- Ivan Shapovalov / intelfx / -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/aa4ed7b0c713aa3e6d8018a1c20bfcb2d4255dab.camel%40intelfx.name.
