Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11520 )
Change subject: IMPALA-7543: Enhance scan ranges to support sub-ranges ...................................................................... IMPALA-7543: Enhance scan ranges to support sub-ranges This commit enhances the ScanRange class to make it possible to only read some smaller parts of the whole ScanRange. This functionality is needed by IMPALA-5843. A sub-range is an offset and length which is located within the scan range. Sub-ranges can be added to a scan range when calling ScanRange::Reset(). If done so, the ScanRange class will only read the parts defined by the sub-ranges. If we have sub-ranges for a cache read then the ScanRange won't enqueue the whole cache buffer (which contains the whole ScanRange), but memcpy() the sub-ranges to IO/client buffers. Smaller refactorings needed to do: * remove scan_range_offset_ from BufferDescriptor * number of bytes read are bookkeeped by ScanRange again Testing: * introduced CacheReaderTestStub to fake cache reads during testing * extended disk-io-mgr-test.cc with sub-ranges Change-Id: Iea26ba386713990f7671aab5a372cf449b8d51e4 Reviewed-on: http://gerrit.cloudera.org:8080/11520 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/runtime/io/CMakeLists.txt A be/src/runtime/io/cache-reader-test-stub.h M be/src/runtime/io/disk-io-mgr-stress.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc D be/src/runtime/io/file-reader.cc M be/src/runtime/io/file-reader.h M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/hdfs-file-reader.h M be/src/runtime/io/local-file-reader.cc M be/src/runtime/io/local-file-reader.h M be/src/runtime/io/request-context.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc 15 files changed, 511 insertions(+), 158 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iea26ba386713990f7671aab5a372cf449b8d51e4 Gerrit-Change-Number: 11520 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
