Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16375
Change subject: IMPALA-10076: Reduce partition level update logs ...................................................................... IMPALA-10076: Reduce partition level update logs Catalogd and the legacy mode coordinator log catalog topic items one by a line. A table can have more than 100K partitions. It's too verbose to log all partition level updates. This patch aggregates the partition level updates of each table and only logs the aggregated info in a line. For instance, here is an aggregated log: Collected 24 partition update(s): 1:HDFS_PARTITION:test.tbl:(p=1,p=2,...,p=24), version=1451, original size=(avg=646, min=646, max=648, sum=15516), compressed size=(avg=478, min=475, max=485, sum=11487) If there are more than 3 partitions, we only show the smallest, the second smallest and the largest partition names. If there are only one partition update, the log format is the same as before: Collected 1 partition update(s): 1:HDFS_PARTITION:test.tbl:p=0, version=1451, original size=648, compressed size=475 To support these, this patch implements a helper class, PartitionMetaSummary, to aggregate these information and get a summary for logging. The compressed size is calculated by BE. So the jni method for NativeAddPendingTopicItem is extended to return the actual compressed size. Tests: - Add unit tests for PartitionMetaSummary. Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/PartitionMetaSummary.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java A fe/src/test/java/org/apache/impala/catalog/PartitionMetaSummaryTest.java 11 files changed, 419 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/16375/1 -- To view, visit http://gerrit.cloudera.org:8080/16375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Gerrit-Change-Number: 16375 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <[email protected]>
