Gabor Kaszab has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13938
Change subject: IMPALA-8600 WIP: Refresh transactional tables ...................................................................... IMPALA-8600 WIP: Refresh transactional tables Refreshing a subset of partitions in a transactional table might lead us to an inconsistent state of that transactional table. As a fix user initiated partition refreshes are no longer allowed on ACID tables. Additionally, a refresh partition Metastore event actually triggers a refresh on the whole ACID table. An optimisation is implemented to check the locally latest table level writeId, fetch the same from HMS and do a refresh only if they don't match. This couldn't be done for partitioned tables as apparently Hive doesn't update the table level writeId if the transactional table is partitioned. Similary, checking the writeId for each partition and refresh only the ones where the writeId is not up to date is not feasible either as there is no writeId update when Hive makes schema changes like adding a column neither on table level or on partition level. So after a adding a column in Hive to a partitioned ACID table and refreshing that table in Impala, still Impala wouldn't see the new column. Hence, I unconditionally refresh the whole table if it's ACID and partitioned. Note, that for non-partitioned ACID tables Hive updates the table level writeId even for schema changes. Change-Id: I1851da22452074dbe253bcdd97145e06c7552cd3 --- M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M tests/custom_cluster/test_event_processing.py 7 files changed, 202 insertions(+), 75 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/13938/1 -- To view, visit http://gerrit.cloudera.org:8080/13938 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1851da22452074dbe253bcdd97145e06c7552cd3 Gerrit-Change-Number: 13938 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab <[email protected]>
