Hello Bharath Vissapragada, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13874
to look at the new patch set (#7).
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local
catalog mode
......................................................................
IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
In LocalCatalog implementation, LocalDb#getTable will always return a
completely loaded table containing all the meta of columns, partitions,
files, etc. It's time consuming if we implement the GET_TABLES
HiveServer2 operation based on this interface, since GET_TABLES only
requires table names, table types and table comments, while this
interface will trigger catalogd to fully load the table meta. It becomes
worse when we do this for all the tables.
This patch introduces a new interface, getTableIfCached, to return a
LocalIncompleteTable object if the corresponding table is unloaded,
which requires no round trips to the catalogd. It's used to boost the
GET_TABLES performance in LocalCatalog mode.
Tests
- Testing in a HMS with 100 dbs and 3000 tables, without this patch it
takes ~2mins in GET_TABLES on a cold started cluster. With this patch,
the time reduces to ~1s.
Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
---
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
A fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M tests/hs2/test_hs2.py
11 files changed, 153 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13874/7
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Bharath Vissapragada <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
