Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21143 )
Change subject: IMPALA-12856: Event processor should ignore processing partition with empty partition values ...................................................................... Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/21143/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21143/3//COMMIT_MSG@7 PS3, Line 7: IMPALA-12856: Event processor should ignore processing partition > IMO, we can use this patch to address the empty partition values from HMS a Thank Venu for digging into the HMS bug! Do we have a HIVE JIRA for this? I think in the Impala side, we need some rework to not trust all responses from external systems, or add some retry if hitting such issues twice is pretty rare. http://gerrit.cloudera.org:8080/#/c/21143/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/21143/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2869 PS4, Line 2869: continue; The only concern is the table metadata would be stale after this, i.e. the partition is not updated in the catalog cache. It might miss some files or still point to deleted files. Without this change, the event-processor is able to recover from the IllegalStateException and invalidate the table to avoid stale metadata. As Venu pointed out that the cause is a race issue in HMS, do you think we can retry fetching the partition to better recover from here? -- To view, visit http://gerrit.cloudera.org:8080/21143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2469930ccd74948325f1723bd8b2bd6aad02d09 Gerrit-Change-Number: 21143 Gerrit-PatchSet: 4 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Thu, 21 Mar 2024 23:43:13 +0000 Gerrit-HasComments: Yes
