Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21478 )
Change subject: IMPALA-13120: Load failed table without need for manual invalidate ...................................................................... IMPALA-13120: Load failed table without need for manual invalidate If the metastore is down when the table load is triggered, catalogd updates a new version of incomplete table with cause as TableLoadingException. On coordinator/impalad, StmtMetadataLoader loadTables that has been waiting for table load to complete, considers the table as loaded. Then, during the analyzer’s table resolve step, for the incomplete table, TableLoadingException (i.e., Could not connect to meta store, failed to load metadata for table and running invalidate metadata for table may resolve this problem) is thrown. Henceforth, any query on the table doesn’t trigger the load since the table is incomplete with TableLoadingException cause. Even though metastore is UP later at some time, queries continue to throw the same exception. It is both misleading and also not required to manually invalidate all such tables. Note: Incomplete table with cause is considered as loaded. This patch tries again to load the table that has previously failed due to metastore connection error(i.e., a recoverable error), when a query involving the table is fired. Idea is to keep track of table object present in db that requires load. On successful/failed load, table object in db is updated. Therefore the tracked table object reference can be compared to table object in db to detect the completion of load. Testing: - Added end-to-end tests Change-Id: Ia882fdd865ef716351be7f1eaf203a9fb04c1c15 Reviewed-on: http://gerrit.cloudera.org:8080/21478 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeIncompleteTable.java M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java M tests/custom_cluster/test_catalog_hms_failures.py 6 files changed, 90 insertions(+), 8 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia882fdd865ef716351be7f1eaf203a9fb04c1c15 Gerrit-Change-Number: 21478 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]>
