Hello Quanlong Huang, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21478
to look at the new patch set (#3).
Change subject: IMPALA-13120: Load failed table without need for manual
invalidate
......................................................................
IMPALA-13120: Load failed table without need for manual invalidate
If the metastore is down when the table load is triggered, catalogd
updates a new version of incomplete table with cause as
TableLoadingException. On coordinator/impalad, StmtMetadataLoader
loadTables that has been waiting for table load to complete,
considers the table as loaded. Then, during the analyzer’s table
resolve step, for the incomplete table, TableLoadingException
(i.e., Could not connect to meta store, failed to load metadata for
table and running invalidate metadata for table may resolve this
problem) is thrown.
Henceforth, any query on the table doesn’t trigger the load since
the table is incomplete with TableLoadingException cause. Even though
metastore is UP later at some time, queries continue to throw
the same exception. It is both misleading and also not required to
manually invalidate all such tables.
Note: Incomplete table with cause is considered as loaded.
This patch tries again to load the table that has previously failed
due to metastore connection error(i.e., a recoverable error), when a
query involving the table is fired. Idea is to keep track of table
object present in db that requires load. On successful/failed load,
table object in db is updated. Therefore the tracked table object
reference can be compared to table object in db to detect the
completion of load.
Testing:
- Added end-to-end tests
Change-Id: Ia882fdd865ef716351be7f1eaf203a9fb04c1c15
---
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M tests/custom_cluster/test_catalog_hms_failures.py
5 files changed, 87 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/21478/3
--
To view, visit http://gerrit.cloudera.org:8080/21478
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia882fdd865ef716351be7f1eaf203a9fb04c1c15
Gerrit-Change-Number: 21478
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]>
