Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20768 )
Change subject: IMPALA-12375: Make DataSource Object persistent ...................................................................... IMPALA-12375: Make DataSource Object persistent DataSource objects are saved in-memory cache in Catalog server. They are not persisted to the HMS. The objects are lost after Catalog server is restarted and user needs to recreate DataSource objects before creating new external DataSource tables. This patch makes DataSource Object persistent by saving DataSource objects as DataConnector objects with type "impalaDataSource" in HMS. Since HMS events for DataConnector are not handled, Catalog server has to refresh DataSource objects when the catalogd becomes active. Note that this feature is not supported for Apache Hive 3.1 and older version. Testing: - Added two end-to-end unit tests with restarting of Catalog server, and catalogd HA failover. These two tests are skipped when USE_APACHE_HIVE is set as true and Apache Hive version is 3.x or older version. - Passed all-build-options-ub2004. - Passed core test. Change-Id: I500a99142bb62ce873e693d573064ad4ffa153ab Reviewed-on: http://gerrit.cloudera.org:8080/20768 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Wenzhe Zhou <[email protected]> --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/common/skip.py M tests/custom_cluster/test_ext_data_sources.py 13 files changed, 343 insertions(+), 17 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved Wenzhe Zhou: Verified -- To view, visit http://gerrit.cloudera.org:8080/20768 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I500a99142bb62ce873e693d573064ad4ffa153ab Gerrit-Change-Number: 20768 Gerrit-PatchSet: 10 Gerrit-Owner: Wenzhe Zhou <[email protected]> Gerrit-Reviewer: Abhishek Rawat <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Wenzhe Zhou <[email protected]>
