Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16375 )
Change subject: IMPALA-10076: Reduce partition level update logs ...................................................................... IMPALA-10076: Reduce partition level update logs Catalogd and the legacy mode coordinator log catalog topic items one by a line. A table can have more than 100K partitions. It's too verbose to log all partition level updates. This patch aggregates the partition level updates of each table and only logs the aggregated info in a line. For instance, here is an aggregated log: Collected 24 partition update(s): 1:HDFS_PARTITION:test.tbl:(p=1,p=2,...,p=24), version=1451, original size=(avg=646, min=646, max=648, sum=15516), compressed size=(avg=478, min=475, max=485, sum=11487) If there are more than 3 partitions, we only show the lexicographically smallest, the second smallest and the largest partition names. If there are only one partition update, the log format is the same as before: Collected 1 partition update(s): 1:HDFS_PARTITION:test.tbl:p=0, version=1451, original size=648, compressed size=475 To support these, this patch implements a helper class, PartitionMetaSummary, to aggregate these information and get a summary for logging. The compressed size is calculated by BE. So the jni method for NativeAddPendingTopicItem is extended to return the actual compressed size. Tests: - Added unit tests for PartitionMetaSummary. - Start the cluster in mixed catalog mode and manually verified the logs. Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Reviewed-on: http://gerrit.cloudera.org:8080/16375 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/PartitionMetaSummary.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java A fe/src/test/java/org/apache/impala/catalog/PartitionMetaSummaryTest.java 11 files changed, 438 insertions(+), 26 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Gerrit-Change-Number: 16375 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
