Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/13215 )
Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and
partition
......................................................................
IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
This happens when tables load metadata from HMS.
Add MetastoreShim functions to support HMS3 only functions.
Add validwriteIdlists to query profile through timeline.
Tests:
Manually tests HMS2 and HMS3, using log files to check
Unit tests against HMS3
ToDo:
WriteId and valid writeIds can be fetched in other time, need
more study on that.
Profile example:
Query Compilation: 5s057ms
- Metadata load started: 63.006ms (63.006ms)
- Metadata load finished. loaded-tables=2/2...: 4s801ms (4s738ms)
- Loaded ValidWriteIdLists:
acid.insert_only_no_partitions:6:9223372036854775807::
acid.insert_only_with_partitions:3:9223372036854775807::
: 4s921ms (120.580ms)
- Analysis finished: 4s929ms (8.013ms)
Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54
Reviewed-on: http://gerrit.cloudera.org:8080/13215
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
---
M common/thrift/CatalogObjects.thrift
A fe/src/compat-hive-2/java/org/apache/hadoop/hive/common/ValidWriteIdList.java
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java
15 files changed, 348 insertions(+), 3 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/13215
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54
Gerrit-Change-Number: 13215
Gerrit-PatchSet: 19
Gerrit-Owner: Yongzhi Chen <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Sudhanshu Arora <[email protected]>
Gerrit-Reviewer: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]>
Gerrit-Reviewer: Yongzhi Chen <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
