Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20145 )
Change subject: IMPALA-12256: Fix DDLs losing create event ids in reloading partitions ...................................................................... Patch Set 6: (3 comments) Thank Fang-Yu's review! http://gerrit.cloudera.org:8080/#/c/20145/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java: http://gerrit.cloudera.org:8080/#/c/20145/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@1326 PS4, Line 1326: lose > nit: Should we use "lose" instead? Done http://gerrit.cloudera.org:8080/#/c/20145/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/20145/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2904 PS4, Line 2904: partBuilder.setCreateEventId(oldPartition.getCreateEventId()); : partBuilder.setLastCompactionId(oldPartition.getLastCompactionId()); > I have 2 questions. 1. The lastRefreshEventId is set below at line 2907. Its source value is fetched at line 2889 which is always higher than the id in the old partition. So I think we don't need to set it here. 2. The compaction id is unrelated to this issue. I just fix it by the way. I think losing this id might cause other issues. But I'm not sure whether we already fix it in other places. I think populating the lastCompactionId is safe. http://gerrit.cloudera.org:8080/#/c/20145/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/20145/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4544 PS4, Line 4544: // if the partition has been created since the event was generated, skip : // the stale event. : boolean isStale = hdfsPartition.getCreateEventId() > eventId; : LOG.info("{} partition {} of table {} since it's create event id {} is {} than " + : "eventid {}", : isStale ? "Not dropping" : "Dropping", : hdfsPartition.getPartitionName(), hdfsTable.getFullName(), : hdfsPartition.getCreateEventId(), isStale ? "higher" : "not higher", : eventId); : return !isStale; : > At https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache Yeah, sure. That would be good for debuggability. -- To view, visit http://gerrit.cloudera.org:8080/20145 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I052faa093bda69fb16db0d424c1478bba103dad9 Gerrit-Change-Number: 20145 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Fang-Yu Rao <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Mon, 03 Jul 2023 08:05:54 +0000 Gerrit-HasComments: Yes
