Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21035
Change subject: IMPALA-12818: Intermediate Result Caching plan node framework ...................................................................... IMPALA-12818: Intermediate Result Caching plan node framework This patch adds a plan node framework for caching of intermediate result tuples within a query. Actual caching of data will be implemented in subsequent patches. A new plan node type TupleCacheNode is introduced for brokering caching decisions at runtime. If the result is in the cache, the TupleCacheNode will return results from the cache and skip executing its child node. If the result is not cached, the TupleCacheNode will execute its child node and mirror the resulting RowBatches to the cache. The TupleCachePlanner decides where to place the TupleCacheNodes. To calculate eligibility and cache keys, the plan must be in a stable state that will not change shape. TupleCachePlanner currently runs at the end of planning after the DistributedPlanner and ParallelPlanner have run. As a first cut, TupleCachePlanner places TupleCacheNodes at every eligible location. Eligibility is currently restricted to immediately above HdfsScanNodes. This implementation will need to incorporate cost heuristics and other policies for placement. Each TupleCacheNode has a hash key that is generated from the logical plan below for the purpose of identifying results that have been cached by semantically equivalent query subtrees. The initial implementation of the subtree hash uses the plan Thrift to uniquely identify the subtree. Tuple caching is enabled by setting the enable_tuple_cache query option to true. As a safeguard during development, enable_tuple_cache can only be set to true if the "allow_tuple_caching" startup option is set to true. It defaults to false to minimize the impact for production clusters. bin/start-impala-cluster.py sets allow_tuple_caching=true by default to enable it in the development environment. Testing: - This adds a frontend test that does basic checks for cache keys and eligibility - This verifies the presence of the caching information in the explain plan output. Change-Id: Ia1f36a87dcce6efd5d1e1f0bc04009bf009b1961 --- M be/src/exec/CMakeLists.txt M be/src/exec/exec-node.cc A be/src/exec/tuple-cache-node.cc A be/src/exec/tuple-cache-node.h M be/src/service/query-options.cc M be/src/service/query-options.h M bin/start-impala-cluster.py M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java A fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java A fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java A fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java A fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java M testdata/workloads/functional-query/queries/QueryTest/explain-level1.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test 19 files changed, 816 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/21035/1 -- To view, visit http://gerrit.cloudera.org:8080/21035 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia1f36a87dcce6efd5d1e1f0bc04009bf009b1961 Gerrit-Change-Number: 21035 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]>
