Wenzhe Zhou has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/20768 )
Change subject: IMPALA-12375: Make DataSource Object persistent ...................................................................... IMPALA-12375: Make DataSource Object persistent DataSource objects are saved in-memory cache in Catalog server. They are not persisted to the HMS. The objects are lost after Catalog server is restarted. This patch makes DataSource Object persistent by saving DataSource objects as DataConnector objects with type "impalaDataSource" in HMS. Since HMS events for DataConnector are not handled, Catalog server has to refresh DataSource objects when the catalogd becomes active. Note that this feature is not supported for Apache Hive 3.1 and older version. Testing: - Added two end-to-end unit tests with restarting of Catalog server, and catalogd HA failover. These two tests are skipped when USE_APACHE_HIVE is set as true and Apache Hive version is 3.x or older version. - Passed all-build-options-ub2004. - Passed core test. Change-Id: I500a99142bb62ce873e693d573064ad4ffa153ab --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/common/skip.py M tests/custom_cluster/test_ext_data_sources.py 13 files changed, 336 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/20768/6 -- To view, visit http://gerrit.cloudera.org:8080/20768 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I500a99142bb62ce873e693d573064ad4ffa153ab Gerrit-Change-Number: 20768 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou <[email protected]> Gerrit-Reviewer: Abhishek Rawat <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]>
