[
https://issues.apache.org/jira/browse/IMPALA-13057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850883#comment-17850883
]
ASF subversion and git services commented on IMPALA-13057:
----------------------------------------------------------
Commit 825900fa6c3a51941b7b90edb8af6f7dba5e5fe8 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=825900fa6 ]
IMPALA-13057: Incorporate tuple/slot information into tuple cache key
The tuple cache keys currently do not include information about
the tuples or slots, as that information is stored outside
the PlanNode thrift structures. The tuple/slot information is
critical to determining which columns are referenced and what
data layout the result tuple has. This adds code to incorporate
the TupleDescriptors and SlotDescriptors into the cache key.
Since the tuple and slot ids are indexes into a global structure
(the descriptor table), they hinder cache key matches across
different queries. If a query has an extra filter, it can shift
all the slot ids. If the query has an extra join, it can
shift all the tuple ids. To eliminate this effect, this adds the
ability to translate tuple and slot ids from global indices to
local indices. The translation only contains information from the
subtree below that point, so it is not influenced by unrelated
parts of the query.
When the code registers a tuple with the TupleCacheInfo, it also
registers a translation from the global index to a local index.
Any code that puts SlotIds or TupleIds into a Thrift data structure
can use the translateTupleId() and translateSlotId() functions to
get the local index. These are exposed on ThriftSerializationCtx
by functions of the same name, but those functions apply the
translation only when working for the tuple cache.
This passes the ThriftSerializationCtx into Exprs that have
TupleIds or SlotIds and applies the translation. It also passes
the ThriftSerializationCtx into PlanNode::toThrift(), which is
used to translate TupleIds in HdfsScanNode.
This also adds a way to register a table with the tuple cache
and incorporate information about it. This allows us to mask
out additional fields in PlanNode and enable a test case that
relies on matching with different table aliases.
Testing:
- This fixes some commented out test cases in TupleCacheTest
(specifically telling columns apart)
- This adds new test cases that match due to id translation
(extra filters, extra joins)
- This adds a unit test for the id translation to
TupleCacheInfoTest
Change-Id: I7f5278e9dbb976cbebdc6a21a6e66bc90ce06c6c
Reviewed-on: http://gerrit.cloudera.org:8080/21398
Reviewed-by: Joe McDonnell <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Incorporate tuple/slot information into the tuple cache key
> -----------------------------------------------------------
>
> Key: IMPALA-13057
> URL: https://issues.apache.org/jira/browse/IMPALA-13057
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.4.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> Since the tuple and slot information is kept separately in the descriptor
> table, it does not get incorporated into the PlanNode thrift used for the
> tuple cache key. This means that the tuple cache can't distinguish between
> these two queries:
> {noformat}
> select int_col1 from table;
> select int_col2 from table;{noformat}
> To solve this, the tuple/slot information needs to be incorporated into the
> cache key. PlanNode::initThrift() walks through each tuple, so this is a good
> place to serialize the TupleDescriptor/SlotDescriptors and incorporate it
> into the hash.
> The tuple ids and slot ids are global ids, so the value is influenced by the
> entirety of the query. This is a problem for matching cache results across
> different queries. As part of incorporating the tuple/slot information, we
> should also add an ability to translate tuple/slot ids into ids local to a
> subtree.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
