Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23228
Change subject: IMPALA-13548: Schedule scan ranges oldest to newest for tuple caching ...................................................................... IMPALA-13548: Schedule scan ranges oldest to newest for tuple caching Scheduling does not sort scan ranges by modification time. When a new file is added to a table, it's order in the list of scan ranges is not based on modification time. Instead, it is based on which partition it belongs to and what its filename is. A new file that is added early in the list of scan ranges can cause cascading differences in scheduling. For tuple caching, this means that multiple runtime cache keys could change due to adding a single file. To minimize that disruption, this adds the ability to sort the scan ranges by modification time and schedule scan ranges oldest to newest. This enables it for scan nodes that feed into tuple cache nodes (similar to deterministic scan range assignment). Testing: - Modified TestTupleCacheFullCluster::test_scan_range_distributed to have stricter checks about how many cache keys change after an insert (only one should change) - Modified TupleCacheTest#testDeterministicScheduling to verify that oldest to newest scheduling is also enabled. Change-Id: Ia4108c7a00c6acf8bbfc036b2b76e7c02ae44d47 --- M be/src/scheduling/scheduler.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java M fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java M tests/custom_cluster/test_tuple_cache.py 6 files changed, 85 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/23228/1 -- To view, visit http://gerrit.cloudera.org:8080/23228 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia4108c7a00c6acf8bbfc036b2b76e7c02ae44d47 Gerrit-Change-Number: 23228 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com>
