Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20673 )
Change subject: IMPALA-12486: Add catalog metrics for metadata loading ...................................................................... IMPALA-12486: Add catalog metrics for metadata loading This patch adds the following catalog metrics which indicate the load on HDFS for loading file metadata: - catalog-server.metadata.file.num-loading-threads: The total size of all thread pools used in loading file metadata. - catalog-server.metadata.file.num-loading-tasks: The total number of unfinished file metadata loading tasks. Each task corresponds to a partition. - catalog-server.metadata.table.num-loading-file-metadata: The total number of tables that are loading file metadata. Also adds some metrics for metadata loading on all tables. Note that metadata loading of an HDFS table consists of loading HMS metadata and HDFS file metadata, etc. - catalog-server.metadata.table.num-loading-metadata: The total number of tables that are loading metadata. - catalog-server.metadata.table.async-loading.num-in-progress: The total number of tables that are loading metadata asynchronously. E.g. the initial metadata loading triggered by the first access on a table. - catalog-server.metadata.table.async-loading.queue-len: The total number of tables that are waiting for asynchronous loading. If this number raises, consider bumping --num_metadata_loading_threads. Three metrics about the catalog cache are also added: - catalog.num-databases - catalog.num-tables - catalog.num-functions Note that the first two are also shown in WebUI of coordinators and we plan to deprecate them and only show them in catalogd's WebUI. The number of idle and in-use HMS clients is also exposed in this patch: - catalog.hms-client-pool.num-idle - catalog.hms-client-pool.num-in-use Tests - Launch catalogd locally with load_catalog_in_background=true and verified the metrics. - Add e2e tests in tests/webserver/test_web_pages.py Change-Id: Icef7b123bdcb0f5b8572635eeaacd8294990f9ba Reviewed-on: http://gerrit.cloudera.org:8080/20673 Reviewed-by: Andrew Sherman <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M common/thrift/JniCatalog.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java M fe/src/main/java/org/apache/impala/catalog/View.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/webserver/test_web_pages.py 21 files changed, 367 insertions(+), 16 deletions(-) Approvals: Andrew Sherman: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/20673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icef7b123bdcb0f5b8572635eeaacd8294990f9ba Gerrit-Change-Number: 20673 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Andrew Sherman <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]>
