Hello Vihang Karajgaonkar, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16375 to look at the new patch set (#3). Change subject: IMPALA-10076: Reduce partition level update logs ...................................................................... IMPALA-10076: Reduce partition level update logs Catalogd and the legacy mode coordinator log catalog topic items one by a line. A table can have more than 100K partitions. It's too verbose to log all partition level updates. This patch aggregates the partition level updates of each table and only logs the aggregated info in a line. For instance, here is an aggregated log: Collected 24 partition update(s): 1:HDFS_PARTITION:test.tbl:(p=1,p=2,...,p=24), version=1451, original size=(avg=646, min=646, max=648, sum=15516), compressed size=(avg=478, min=475, max=485, sum=11487) If there are more than 3 partitions, we only show the smallest, the second smallest and the largest partition names. If there are only one partition update, the log format is the same as before: Collected 1 partition update(s): 1:HDFS_PARTITION:test.tbl:p=0, version=1451, original size=648, compressed size=475 To support these, this patch implements a helper class, PartitionMetaSummary, to aggregate these information and get a summary for logging. The compressed size is calculated by BE. So the jni method for NativeAddPendingTopicItem is extended to return the actual compressed size. Tests: - Added unit tests for PartitionMetaSummary. - Start the cluster in mixed catalog mode and manually verified the logs. Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/PartitionMetaSummary.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java A fe/src/test/java/org/apache/impala/catalog/PartitionMetaSummaryTest.java 11 files changed, 432 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/16375/3 -- To view, visit http://gerrit.cloudera.org:8080/16375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Gerrit-Change-Number: 16375 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>