Internal Jenkins has submitted this change and it was merged. Change subject: IMPALA-2347: Reuse metastore client connections in Catalog ......................................................................
IMPALA-2347: Reuse metastore client connections in Catalog Currently we create a new connection to metastore every time Catalog connects to HMS. This was intentionally done to circumvent HIVE-5181. Given it is fixed already in Hive, this patch intends to refactor the HMS client usage on the catalog to reuse the connections. Additionally this patch makes MetaStoreClient implement AutoCloseable interface and hence all the callers can use the try-with-resources to create a new metastore client and needn't explicitly call release(). Also, this patch increases the default initial metastore pool size to 10 from a previous value of 5, which is less even for a decent DDL load. In terms of design, this patch switches the metastore client implementation to RetryingMetaStoreClient from previous implementation of HiveMetaStoreClient. The reason for this switch is to handle HMS failures from Catalog side where the entire metastore client pool cache becomes stale in the event of a metastore restart and there is no proper way to deal with it. RetryingMetaStoreClient has inbuilt retry mechanism which reconnects stale connections in the event of failures. For more details on retries and corresponding configurations, check org.apache.hadoop.hive.metastore.RetryingMetaStoreClient. Change-Id: I517c0e1efef2584cd8d34017b33574f2ad69bd52 Reviewed-on: http://gerrit.cloudera.org:8080/3984 Reviewed-by: Bharath Vissapragada <[email protected]> Tested-by: Internal Jenkins --- M fe/src/main/java/com/cloudera/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java M fe/src/main/java/com/cloudera/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/com/cloudera/impala/catalog/DataSourceTable.java M fe/src/main/java/com/cloudera/impala/catalog/HBaseTable.java M fe/src/main/java/com/cloudera/impala/catalog/HdfsTable.java M fe/src/main/java/com/cloudera/impala/catalog/ImpaladCatalog.java M fe/src/main/java/com/cloudera/impala/catalog/IncompleteTable.java M fe/src/main/java/com/cloudera/impala/catalog/KuduTable.java M fe/src/main/java/com/cloudera/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/com/cloudera/impala/catalog/Table.java M fe/src/main/java/com/cloudera/impala/catalog/TableLoader.java M fe/src/main/java/com/cloudera/impala/catalog/View.java M fe/src/main/java/com/cloudera/impala/service/CatalogOpExecutor.java M fe/src/main/java/com/cloudera/impala/util/MetaStoreUtil.java M fe/src/test/java/com/cloudera/impala/catalog/CatalogTest.java 16 files changed, 124 insertions(+), 190 deletions(-) Approvals: Bharath Vissapragada: Looks good to me, approved Internal Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/3984 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I517c0e1efef2584cd8d34017b33574f2ad69bd52 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Internal Jenkins
