Hello Quanlong Huang, Fang-Yu Rao, Steve Carlin, Michael Smith, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23436
to look at the new patch set (#5).
Change subject: IMPALA-14447: Parallelize table loading in getMissingTables()
......................................................................
IMPALA-14447: Parallelize table loading in getMissingTables()
StmtMetadataLoader.getMissingTables() load missing tables in serial
manner. In local catalog mode, large number of serial table loading can
incur significant round trip latency to CatalogD. This patch improve the
table loading by using parallelStream to lookup and gather all non-null
FeTables from given TableName set.
Modify LocalCatalog.loadDbs() slightly to make it thread-safe. Change
FrontendProfile.Scope to support nested scope referencing the same
FrontendProfile instance.
Testing:
Run and pass few tests such as test_catalogd_ha.py,
test_concurrent_ddls.py, and test_observability.py.
Add FE tests CatalogdMetaProviderTest.testProfileParallelLoad.
Manually run following query and observe parallel loading by setting
TRACE level log in CatalogdMetaProvider.java.
use functional;
select count(*) from alltypesnopart
union select count(*) from alltypessmall
union select count(*) from alltypestiny
union select count(*) from alltypesagg;
Change-Id: I97a5165844ae846b28338d62e93a20121488d79f
---
M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/service/FrontendProfile.java
M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
4 files changed, 136 insertions(+), 35 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/23436/5
--
To view, visit http://gerrit.cloudera.org:8080/23436
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I97a5165844ae846b28338d62e93a20121488d79f
Gerrit-Change-Number: 23436
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <riza.sumi...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fangyu....@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <scar...@cloudera.com>
