[
https://issues.apache.org/jira/browse/IMPALA-8675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17768768#comment-17768768
]
ASF subversion and git services commented on IMPALA-8675:
---------------------------------------------------------
Commit 6af8154ecb7795506d1afebf044c89cf2cf83f8e in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=6af8154ec ]
IMPALA-8675: Remove db/table count metrics from impalad in LocalCatalog mode
In the /metrics webUI, coordinator shows metrics of
"catalog.num-databases" and "catalog.num-tables" for its local catalog
cache. They are updated at the end of each query execution, via
Frontend.getCatalogMetrics().
In LocalCatalog mode, there is no need for every coordinator to have the
full list of tables of every database. However, getCatalogMetrics ends
up iterating over every DB and fetching these lists (if uncached) in
order to provide a count. This introduces unnecessary catalog RPCs at
the end of each query execution. When catalogd is slow/hanging in
processing such coordinator RPCs, simple queries will also be hanging.
This patch removes tracking the db/table count metrics from coordinator
side in LocalCatalog mode. They will always be -1. The count isn't
particularly relevant – if someone wants to keep track of the size of
their catalog they are better off looking at that metric from catalogd.
Tests:
- test_non_compact_catalog_topic_updates uses these two metrics to
detect new catalog updates. Changed it to use "catalog.curr-version"
instead.
Change-Id: I02a409b7b24577f75d7c439c85bc3491ec7c518c
Reviewed-on: http://gerrit.cloudera.org:8080/20500
Reviewed-by: Wenzhe Zhou <[email protected]>
Tested-by: Wenzhe Zhou <[email protected]>
> Remove db/table count metrics from impalad in LocalCatalog mode
> ---------------------------------------------------------------
>
> Key: IMPALA-8675
> URL: https://issues.apache.org/jira/browse/IMPALA-8675
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Todd Lipcon
> Assignee: Quanlong Huang
> Priority: Minor
> Labels: catalog-v2
>
> In LocalCatalog there is no need for every coordinator to have the full list
> of tables of every database. But, getCatalogMetrics ends up iterating over
> every DB and fetching these lists in order to provide a count. The count
> isn't particularly relevant -- if someone wants to keep track of the size of
> their catalog they are better off looking at that metric from catalogd. We
> should remove these catalog metrics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
