Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24989 )
Change subject: IMPALA-14778: Fix Close() crashes in Iceberg metadata and Paimon scans ...................................................................... IMPALA-14778: Fix Close() crashes in Iceberg metadata and Paimon scans IcebergMetadataScanNode::Close() crashed impalad when Open() did not complete: - If the scanner was never created, e.g. because a UNION closed an unopened child after a cancellation or after reaching its LIMIT, Close() dereferenced a null 'metadata_scanner_'. - If the base table was dropped, failed to load or was no longer an Iceberg table when Open() reloaded it, the Java scanner's constructor threw, and Close() passed the uninitialized 'jmetadata_scanner_' to DeleteGlobalRef(). This patch initializes the pointer and checks the scanner in Close(). JniFrontend.getCatalogTable() throws TableNotFoundException instead of returning null, and IcebergMetadataScanner checks the table, so these cases fail with "Cannot scan metadata table ..." and the load error as the cause. IcebergMetadataScanNode::GetNext() now calls ExecDebugAction() like other scan nodes. PaimonJniScanner and PaimonJniScanNode::Close() had the same two bugs and are fixed the same way. PaimonJniScanner::Close() now calls the Java close() through JniCall, so an exception from it is logged instead of being left pending on the thread. IcebergMetadataScanner also leaked the JNI global reference to the base table, and the Java scanner never closed its Iceberg iterators, leaving manifest readers open, e.g. for ENTRIES. Close() now deletes the reference and calls the new Java close(), and the rows iterator of each task is closed when the scan moves to the next task. Testing: - Added EE tests for the crashing cases, a metadata table query to test_failpoints.py, and FE tests. - Checked manually that 'JNI global refs' (jcmd Thread.print) no longer grows with metadata table queries. Assisted-by: Claude Opus 5.5 <[email protected]> Change-Id: I0519035ecdca966bdc1a26147a157dc62bad2d8f Reviewed-on: http://gerrit.cloudera.org:8080/24989 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc M be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.h M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.cc M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.h M be/src/exec/paimon/paimon-jni-scan-node.cc M be/src/exec/paimon/paimon-jni-scan-node.h M be/src/exec/paimon/paimon-jni-scanner.cc M be/src/exec/paimon/paimon-jni-scanner.h M be/src/service/frontend.h M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java A fe/src/test/java/org/apache/impala/util/IcebergMetadataScannerTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test M testdata/workloads/functional-query/queries/QueryTest/paimon-query.test M tests/failure/test_failpoints.py M tests/query_test/test_iceberg.py 16 files changed, 403 insertions(+), 31 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24989 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0519035ecdca966bdc1a26147a157dc62bad2d8f Gerrit-Change-Number: 24989 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
