Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21398 )
Change subject: IMPALA-13057: Incorporate tuple/slot information into tuple cache key ...................................................................... IMPALA-13057: Incorporate tuple/slot information into tuple cache key The tuple cache keys currently do not include information about the tuples or slots, as that information is stored outside the PlanNode thrift structures. The tuple/slot information is critical to determining which columns are referenced and what data layout the result tuple has. This adds code to incorporate the TupleDescriptors and SlotDescriptors into the cache key. Since the tuple and slot ids are indexes into a global structure (the descriptor table), they hinder cache key matches across different queries. If a query has an extra filter, it can shift all the slot ids. If the query has an extra join, it can shift all the tuple ids. To eliminate this effect, this adds the ability to translate tuple and slot ids from global indices to local indices. The translation only contains information from the subtree below that point, so it is not influenced by unrelated parts of the query. When the code registers a tuple with the TupleCacheInfo, it also registers a translation from the global index to a local index. Any code that puts SlotIds or TupleIds into a Thrift data structure can use the translateTupleId() and translateSlotId() functions to get the local index. These are exposed on ThriftSerializationCtx by functions of the same name, but those functions apply the translation only when working for the tuple cache. This passes the ThriftSerializationCtx into Exprs that have TupleIds or SlotIds and applies the translation. It also passes the ThriftSerializationCtx into PlanNode::toThrift(), which is used to translate TupleIds in HdfsScanNode. This also adds a way to register a table with the tuple cache and incorporate information about it. This allows us to mask out additional fields in PlanNode and enable a test case that relies on matching with different table aliases. Testing: - This fixes some commented out test cases in TupleCacheTest (specifically telling columns apart) - This adds new test cases that match due to id translation (extra filters, extra joins) - This adds a unit test for the id translation to TupleCacheInfoTest Change-Id: I7f5278e9dbb976cbebdc6a21a6e66bc90ce06c6c Reviewed-on: http://gerrit.cloudera.org:8080/21398 Reviewed-by: Joe McDonnell <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/analysis/CastExpr.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java M fe/src/main/java/org/apache/impala/common/ThriftSerializationCtx.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java M fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java M fe/src/test/java/org/apache/impala/planner/TupleCacheInfoTest.java M fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java M testdata/workloads/functional-query/queries/QueryTest/explain-level1.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test 17 files changed, 504 insertions(+), 75 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/21398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7f5278e9dbb976cbebdc6a21a6e66bc90ce06c6c Gerrit-Change-Number: 21398 Gerrit-PatchSet: 8 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]>
