Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21035 )
Change subject: IMPALA-12818: Intermediate Result Caching plan node framework ...................................................................... IMPALA-12818: Intermediate Result Caching plan node framework This patch adds a plan node framework for caching of intermediate result tuples within a query. Actual caching of data will be implemented in subsequent patches. A new plan node type TupleCacheNode is introduced for brokering caching decisions at runtime. If the result is in the cache, the TupleCacheNode will return results from the cache and skip executing its child node. If the result is not cached, the TupleCacheNode will execute its child node and mirror the resulting RowBatches to the cache. The TupleCachePlanner decides where to place the TupleCacheNodes. To calculate eligibility and cache keys, the plan must be in a stable state that will not change shape. TupleCachePlanner currently runs at the end of planning after the DistributedPlanner and ParallelPlanner have run. As a first cut, TupleCachePlanner places TupleCacheNodes at every eligible location. Eligibility is currently restricted to immediately above HdfsScanNodes. This implementation will need to incorporate cost heuristics and other policies for placement. Each TupleCacheNode has a hash key that is generated from the logical plan below for the purpose of identifying results that have been cached by semantically equivalent query subtrees. The initial implementation of the subtree hash uses the plan Thrift to uniquely identify the subtree. Tuple caching is enabled by setting the enable_tuple_cache query option to true. As a safeguard during development, enable_tuple_cache can only be set to true if the "allow_tuple_caching" startup option is set to true. It defaults to false to minimize the impact for production clusters. bin/start-impala-cluster.py sets allow_tuple_caching=true by default to enable it in the development environment. Testing: - This adds a frontend test that does basic checks for cache keys and eligibility - This verifies the presence of the caching information in the explain plan output. Change-Id: Ia1f36a87dcce6efd5d1e1f0bc04009bf009b1961 Reviewed-on: http://gerrit.cloudera.org:8080/21035 Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Michael Smith <[email protected]> Reviewed-by: Yida Wu <[email protected]> Reviewed-by: Kurt Deschler <[email protected]> --- M be/src/exec/CMakeLists.txt M be/src/exec/exec-node.cc A be/src/exec/tuple-cache-node.cc A be/src/exec/tuple-cache-node.h M be/src/service/query-options.cc M be/src/service/query-options.h M bin/start-impala-cluster.py M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift M common/thrift/Query.thrift A fe/src/main/java/org/apache/impala/common/ThriftSerializationCtx.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java A fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java A fe/src/main/java/org/apache/impala/planner/TupleCacheNode.java A fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java A fe/src/test/java/org/apache/impala/planner/TupleCacheInfoTest.java A fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java M testdata/workloads/functional-query/queries/QueryTest/explain-level1.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test 21 files changed, 1,124 insertions(+), 16 deletions(-) Approvals: Impala Public Jenkins: Verified Michael Smith: Looks good to me, but someone else must approve Yida Wu: Looks good to me, but someone else must approve Kurt Deschler: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/21035 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia1f36a87dcce6efd5d1e1f0bc04009bf009b1961 Gerrit-Change-Number: 21035 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]>
