Hello Arnab Karmakar, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24949
to look at the new patch set (#2).
Change subject: IMPALA-15423: Fix non-ASCII strings in Iceberg metadata tables
......................................................................
IMPALA-15423: Fix non-ASCII strings in Iceberg metadata tables
IcebergRowReader copied strings from the JVM with JniUtfCharGuard,
whose size was the number of UTF-16 code units of the string, while
its buffer held the modified UTF-8 encoding. Non-ASCII characters
take more bytes than code units, so the strings were truncated, e.g.
'Zürich' was returned as 'Züric'. Modified UTF-8 also differs from
standard UTF-8 for supplementary characters and NUL.
The metadata scanner also leaked JNI local references: the byte
arrays of string and binary values, and the keys and values of map
entries were never deleted. The fragment thread is attached to the
JVM, so these references kept the objects alive until the thread
detached, i.e. for the whole scan.
This patch:
- converts strings to UTF-8 byte arrays on the Java side and copies
them with JniByteArrayGuard,
- deletes these local references after use, and pushes a local frame
for each row, so the remaining references of a row are freed too,
- makes JniUtfCharGuard::get_size() return the size of its buffer.
None of its remaining callers use it.
Testing:
- Added EE tests that read partition values with non-ASCII
characters (including a supplementary character) and with NUL from
the files and partitions metadata tables. They failed without the
fix.
- Checked with a class histogram of the coordinator JVM that a scan
of the files table of a table with 1824 data files no longer keeps
the map values alive until the end of the scan.
- Ran test_iceberg.py.
Change-Id: I862e327e4b7b93dfabb660d88a21e04ee7b03cd5
Assisted-by: Claude Opus 5.5 (1M context) <[email protected]>
---
M be/src/exec/iceberg-metadata/iceberg-metadata-scan-node.cc
M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.cc
M be/src/exec/iceberg-metadata/iceberg-metadata-scanner.h
M be/src/exec/iceberg-metadata/iceberg-row-reader.cc
M be/src/exec/iceberg-metadata/iceberg-row-reader.h
M be/src/util/jni-util.cc
M be/src/util/jni-util.h
M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-metadata-tables.test
9 files changed, 110 insertions(+), 36 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24949/2
--
To view, visit http://gerrit.cloudera.org:8080/24949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I862e327e4b7b93dfabb660d88a21e04ee7b03cd5
Gerrit-Change-Number: 24949
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
