Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18926 )
Change subject: IMPALA-11540: Add logs for ALTER_TABLE events that trigger slow metadata reload ...................................................................... Patch Set 3: (2 comments) > Patch Set 3: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8542/ The failure is due to compilation error on Apache Hive 3.x: https://jenkins.impala.io/job/all-build-options-ub1604/8722/console http://gerrit.cloudera.org:8080/#/c/18926/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/18926/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1462 PS3, Line 1462: long duration = System.currentTimeMillis() - start_ms; > Hi quanlong, System.nanoTime() is more accurate than System.currentTimeMill Done http://gerrit.cloudera.org:8080/#/c/18926/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1474 PS3, Line 1474: if (tableBefore_.isSetColStats() || tableAfter_.isSetColStats()) { The colStats field is added in HIVE-21078 which is not in Hive-3.1.2. This causes compilation error when building with Apache Hive 3.x. In my experiments, the colStats is always null in the AlterTable events. It seems it's not set into the event, at least for the oldTable object: https://github.com/apache/hive/blob/4e4e39c471094567dcdfd9840edbd99d7eafc230/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L6117 https://github.infra.cloudera.com/CDH/hive/blob/64234e1217b105ac74216e7877a75a14b141ee6d/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L3733 Hive also have codes logging the Table#toString() directly, e.g. https://github.com/apache/hive/blob/4e4e39c471094567dcdfd9840edbd99d7eafc230/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L708 https://github.com/apache/hive/blob/4e4e39c471094567dcdfd9840edbd99d7eafc230/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L766 BTW, I think even if we log the stats, the complexity is the same as logging the column schema. There are no large binaries like incremental stats of Impala. So I think it's ok to ignore these checks. -- To view, visit http://gerrit.cloudera.org:8080/18926 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf344e6b423f88c9635ca8d61d53385b88ba4dce Gerrit-Change-Number: 18926 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Xiang Yang <[email protected]> Gerrit-Reviewer: Yu-Wen Lai <[email protected]> Gerrit-Comment-Date: Thu, 08 Sep 2022 07:42:06 +0000 Gerrit-HasComments: Yes
