Bharath Vissapragada has uploaded a new change for review. http://gerrit.cloudera.org:8080/3984
Change subject: IMPALA-2347: Reuse metastore client connections in Catalog ...................................................................... IMPALA-2347: Reuse metastore client connections in Catalog Currently we create a new connection to metastore every time Catalog connects to HMS. This was intentionally done to circumvent HIVE-5181. Given it is fixed already in Hive, this patch intends to refactor the HMS client usage on the catalog to reuse the connections. Additionally this patch makes MetaStoreClient implement AutoCloseable interface and hence all the callers can use the try-with-resources to create a new metastore client and needn't explicitly call release(). Also, this patch increases the default initial metastore pool size to 10 from a previous value of 5, which is less even for a decent DDL load. In terms of design, this patch switches the metastore client implementation to RetryingMetaStoreClient from previous implementation of HiveMetaStoreClient. The reason for this switch is to handle HMS failures from Catalog side where the entire metastore client pool cache becomes stale in the event of a metastore restart and there is no proper way to deal with it. RetryingMetaStoreClient has inbuilt retry mechanism which reconnects stale connections in the event of failures. For more details on retries and corresponding configurations, check org.apache.hadoop.hive.metastore.RetryingMetaStoreClient. Change-Id: I517c0e1efef2584cd8d34017b33574f2ad69bd52 --- M fe/src/main/java/com/cloudera/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java M fe/src/main/java/com/cloudera/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/com/cloudera/impala/catalog/DataSourceTable.java M fe/src/main/java/com/cloudera/impala/catalog/HBaseTable.java M fe/src/main/java/com/cloudera/impala/catalog/HdfsTable.java M fe/src/main/java/com/cloudera/impala/catalog/ImpaladCatalog.java M fe/src/main/java/com/cloudera/impala/catalog/IncompleteTable.java M fe/src/main/java/com/cloudera/impala/catalog/KuduTable.java M fe/src/main/java/com/cloudera/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/com/cloudera/impala/catalog/Table.java M fe/src/main/java/com/cloudera/impala/catalog/TableLoader.java M fe/src/main/java/com/cloudera/impala/catalog/View.java M fe/src/main/java/com/cloudera/impala/service/CatalogOpExecutor.java M fe/src/main/java/com/cloudera/impala/util/MetaStoreUtil.java M fe/src/test/java/com/cloudera/impala/catalog/CatalogTest.java 16 files changed, 124 insertions(+), 190 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/3984/1 -- To view, visit http://gerrit.cloudera.org:8080/3984 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I517c0e1efef2584cd8d34017b33574f2ad69bd52 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada <[email protected]>
