Hello Lars Volker, Zoltan Borok-Nagy,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10116
to look at the new patch set (#6).
Change subject: IMPALA-6131: Track time of last statistics update in metadata
......................................................................
IMPALA-6131: Track time of last statistics update in metadata
The timestamp of the last COMPUTE STATS operation is saved to
table property "impala.lastComputeStatsTime". The format is
the same as in "transient_lastDdlTime", so the two can be
compared to check if the schema has changed since computing
statistics.
Other changes:
- Handling of "transient_lastDdlTime" is simplified - the old
logic set it to current time + 1, if the old version was
>= current time, to ensure that it is always increased by
DDL operations. This was useful in the past, as IMPALA-387
used lastDdlTime to check if partition data needs to be
reloaded, but since IMPALA-1480, Impala does not rely on
lastDdlTime at all.
- Computing / setting stats on HDFS tables no longer increases
"transient_lastDdlTime".
- When Kudu tables are (re)loaded, it is checked if their
HMS representation is up to date, and if it is, then
IMetaStoreClient.alter_table() is not called. The old
logic always called alter_table() after loading metadata
from Kudu. This change was needed to ensure that
"transient_lastDdlTime" works similarly in HDFS and Kudu
tables, and should also make (re)loading Kudu tables faster.
Notes:
- Kudu will be able to sync its tables to HMS in the near
future (see KUDU-2191), so the Kudu metadata handling in
Impala may need to be redesigned.
Testing:
tests/metadata/test_last_ddl_time_update.py is extended by
- also checking "impala.lastComputeStatsTime"
- testing more SQL statements
- tests for Kudu tables
Note that test_last_ddl_time_update.py is ran only in
exhaustive testing.
Change-Id: I59a671ac29d352bd92ce40d5cb6662bb23f146b5
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/metadata/test_last_ddl_time_update.py
6 files changed, 182 insertions(+), 152 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/10116/6
--
To view, visit http://gerrit.cloudera.org:8080/10116
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I59a671ac29d352bd92ce40d5cb6662bb23f146b5
Gerrit-Change-Number: 10116
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Lars Volker <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
