Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23074
Change subject: IMPALA-13898: Incorporate partition information into tuple cache keys ...................................................................... IMPALA-13898: Incorporate partition information into tuple cache keys Currently, the tuple cache keys do not include partition information in either the planner key or the fragment instance key. However, the partition actually is important to correctness. First, there are settings defined on the table and partition that can impact the results. For example, for processing text files, the separator, escape character, etc are specified at the table level. This impacts the rows produced from a given file. There are other such settings stored at the partition level (e.g. the JSON binary format). Second, it is possible to have two partitions pointed at the same filesystem location. For example, scale_db.num_partitions_1234_blocks_per_partition_1 is a table that has all partitions pointing to the same location. In that case, the cache can't tell the partitions apart based on the files alone. This is an exotic configuration. Incorporating an identifier of the partition (e.g. the partition keys/values) allows the cache to tell the difference. To fix this, we incorporate partition information into the key. At planning time, when incorporating the scan range information, we also incorporate information about the associated partitions. This moves the code to HdfsScanNode and changes it to iterate over the partitions, hashing both the partition information and the scan ranges. At runtime, the TupleCacheNode looks up the partition associated with a scan node and hashes the additional information on the HdfsPartitionDescriptor. Testing: - Added a frontend test in TupleCacheTest for a table with multiple partitions pointed at the same place. - Added custom cluster tests testing both issues Change-Id: I3a7109fcf8a30bf915bb566f7d642f8037793a8c --- M be/src/exec/tuple-cache-node.cc M be/src/exec/tuple-cache-node.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java M fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java M tests/common/impala_test_suite.py M tests/custom_cluster/test_tuple_cache.py 9 files changed, 245 insertions(+), 54 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/23074/1 -- To view, visit http://gerrit.cloudera.org:8080/23074 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3a7109fcf8a30bf915bb566f7d642f8037793a8c Gerrit-Change-Number: 23074 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell <[email protected]>
