clintropolis commented on PR #19683: URL: https://github.com/apache/druid/pull/19683#issuecomment-4974071853
>Question 1: Per-historical, do we track in-flight range requests so we avoid duplicate calls? E.g. query 1 calls for range A, not available locally so we initiate fetch from S3, meanwhile query 2 calls for range A. Does query 2 know about the in-flight request to avoid sending a duplicate? yes, though sort of indirectly, there is a set of locks associated with each internal file of the segment which dedupe calls to the same range, so one caller does the work and the others just wait and pick up the result. These locks are acquired in a stable order so that competing callers dont lock each other out. The locking isn't new in this PR, the changes here are more around efficiency/optimization. >Question 2: how are we balancing coalescing ranged GET calls while maximizing parallelism at the S3 connection level? this is tunable, `virtualStorageCoalesceGapBytes` defaults to 1MiB and controls the coalescing part (combining multiple small requests into 1 request) while `virtualStorageMaxFetchRunBytes` defaults to 64MiB and defines when we consider splitting ranges into parallel calls (splits are aligned with internal file boundaries). The default tuning aims for chunk download walltime of ~1 second. I'm not sure how good these defaults are, still need to do some testing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
