Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23228
to look at the new patch set (#2).
Change subject: IMPALA-13548: Schedule scan ranges oldest to newest for tuple
caching
......................................................................
IMPALA-13548: Schedule scan ranges oldest to newest for tuple caching
Scheduling does not sort scan ranges by modification time. When a new
file is added to a table, it's order in the list of scan ranges is
not based on modification time. Instead, it is based on which partition
it belongs to and what its filename is. A new file that is added early
in the list of scan ranges can cause cascading differences in scheduling.
For tuple caching, this means that multiple runtime cache keys could
change due to adding a single file.
To minimize that disruption, this adds the ability to sort the scan
ranges by modification time and schedule scan ranges oldest to newest.
This enables it for scan nodes that feed into tuple cache nodes
(similar to deterministic scan range assignment).
Testing:
- Modified TestTupleCacheFullCluster::test_scan_range_distributed
to have stricter checks about how many cache keys change after
an insert (only one should change)
- Modified TupleCacheTest#testDeterministicScheduling to verify that
oldest to newest scheduling is also enabled.
Change-Id: Ia4108c7a00c6acf8bbfc036b2b76e7c02ae44d47
---
M be/src/scheduling/scheduler.cc
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java
M fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java
M tests/custom_cluster/test_tuple_cache.py
6 files changed, 85 insertions(+), 26 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/23228/2
--
To view, visit http://gerrit.cloudera.org:8080/23228
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia4108c7a00c6acf8bbfc036b2b76e7c02ae44d47
Gerrit-Change-Number: 23228
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
