Hello Vihang Karajgaonkar, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16375
to look at the new patch set (#4).
Change subject: IMPALA-10076: Reduce partition level update logs
......................................................................
IMPALA-10076: Reduce partition level update logs
Catalogd and the legacy mode coordinator log catalog topic items one by
a line. A table can have more than 100K partitions. It's too verbose to
log all partition level updates. This patch aggregates the partition
level updates of each table and only logs the aggregated info in a line.
For instance, here is an aggregated log:
Collected 24 partition update(s):
1:HDFS_PARTITION:test.tbl:(p=1,p=2,...,p=24), version=1451, original
size=(avg=646, min=646, max=648, sum=15516), compressed size=(avg=478,
min=475, max=485, sum=11487)
If there are more than 3 partitions, we only show the lexicographically
smallest, the second smallest and the largest partition names. If there
are only one partition update, the log format is the same as before:
Collected 1 partition update(s): 1:HDFS_PARTITION:test.tbl:p=0,
version=1451, original size=648, compressed size=475
To support these, this patch implements a helper class,
PartitionMetaSummary, to aggregate these information and get a summary
for logging. The compressed size is calculated by BE. So the jni method
for NativeAddPendingTopicItem is extended to return the actual
compressed size.
Tests:
- Added unit tests for PartitionMetaSummary.
- Start the cluster in mixed catalog mode and manually verified the
logs.
Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/service/fe-support.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
A fe/src/main/java/org/apache/impala/catalog/PartitionMetaSummary.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/FeSupport.java
A fe/src/test/java/org/apache/impala/catalog/PartitionMetaSummaryTest.java
11 files changed, 438 insertions(+), 26 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/16375/4
--
To view, visit http://gerrit.cloudera.org:8080/16375
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368
Gerrit-Change-Number: 16375
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]>
