Peter Rozsa has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24949 )
Change subject: IMPALA-15423: Fix non-ASCII strings in Iceberg metadata tables ...................................................................... IMPALA-15423: Fix non-ASCII strings in Iceberg metadata tables IcebergRowReader copied strings from the JVM with JniUtfCharGuard, whose size was the number of UTF-16 code units of the string, while its buffer held the modified UTF-8 encoding. Non-ASCII characters take more bytes than code units, so the strings were truncated, e.g. 'Zürich' was returned as 'Züric'. Modified UTF-8 also differs from standard UTF-8 for supplementary characters and NUL. The metadata scanner also leaked JNI local references: the byte arrays of string and binary values, and the keys and values of map entries were never deleted. The fragment thread is attached to the JVM, so these references kept the objects alive until the thread detached, i.e. for the whole scan. This patch: - converts strings to UTF-8 byte arrays on the Java side and copies them with JniByteArrayGuard, - deletes these local references after use, and pushes a local frame for each row, so the remaining references of a row are freed too, - makes JniUtfCharGuard::get_size() return the size of its buffer. None of its remaining callers use it. Testing: - Added EE tests that read partition values with non-ASCII characters (including a supplementary character) and with NUL from the files and partitions metadata tables. They failed without the fix. - Checked with a class histogram of the coordinator JVM that a scan of the files table of a table with 1824 data files no longer keeps the map values alive until the end of the scan. - Ran test_iceberg.py. Change-Id: I862e327e4b7b93dfabb660d88a21e04ee7b03cd5 Assisted-by: Claude Opus 5.5 (1M context) <[email protected]> Reviewed-on: http://gerrit.cloudera.org:8080/24949 Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Peter Rozsa <[email protected]> --- M be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.cc M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.h M be/src/exec/iceberg-metadata/iceberg-row-reader.cc M be/src/exec/iceberg-metadata/iceberg-row-reader.h M be/src/util/jni-util.cc M be/src/util/jni-util.h M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test 9 files changed, 110 insertions(+), 36 deletions(-) Approvals: Impala Public Jenkins: Verified Peter Rozsa: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/24949 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I862e327e4b7b93dfabb660d88a21e04ee7b03cd5 Gerrit-Change-Number: 24949 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Arnab Karmakar <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
