Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode ...................................................................... IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode In LocalCatalog implementation, LocalDb#getTable will always return a completely loaded table containing all the meta of columns, partitions, files, etc. It's time consuming if we implement the GET_TABLES HiveServer2 operation based on this interface, since GET_TABLES only requires table names, table types and table comments, while this interface will trigger catalogd to fully load the table meta. It becomes worse when we do this for all the tables. This patch introduces a new interface, getTableIfCached, to return a LocalIncompleteTable object if the corresponding table is unloaded, which requires no round trips to the catalogd. It's used to boost the GET_TABLES performance in LocalCatalog mode. Tests - Testing in a HMS with 100 dbs and 3000 tables, without this patch it takes ~2mins in GET_TABLES for all tables on a cold started cluster. With this patch, the time reduces to ~1s. - Testing in HUE-4.4.0 with a db with 3000 tables, the performance is the same as using legacy catalog implementation. Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb Reviewed-on: http://gerrit.cloudera.org:8080/13874 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java A fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M fe/src/test/java/org/apache/impala/service/JdbcTest.java M tests/hs2/test_hs2.py 11 files changed, 156 insertions(+), 15 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13874 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb Gerrit-Change-Number: 13874 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Bharath Vissapragada <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]>
