hubgeter opened a new pull request, #69889:
URL: https://github.com/apache/doris/pull/69889
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
JNI `NewStringUTF` and `GetStringUTFChars` use modified UTF-8, while native
scanner/writer parameters and paths use standard UTF-8. Passing a SQL predicate
containing an emoji to `NewStringUTF` can corrupt or truncate it before the
JDBC driver receives it. Returned maps also encode supplementary characters
differently and cannot preserve a standard UTF-8 round trip.
- Add validated standard UTF-8 ↔ UTF-16 conversion helpers and use them for
both directions of parameter/statistics maps. Preserve supplementary characters
and embedded NULs; reject malformed UTF-8 and unpaired UTF-16 surrogates with
an input offset.
- Migrate Paimon writer text, column/directory arrays, spill-path callbacks,
and exception messages to these helpers. Reuse one array builder and preserve
native spill-path matching and accounting.
- Use standard UTF-8 for dynamic UDF cache signatures and plugin status JSON
as well.
- Fix the existing `String<Global>::new_string` template instantiation and
keep the legacy modified UTF-8 APIs' encoding contract.
### Release note
Fix Unicode corruption in JNI parameters, returned maps, and Paimon metadata
and spill paths. Invalid text encodings now fail explicitly.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Validation: Focused ASAN_UT build and execution via `./run-be-ut.sh --run
--filter='JniUtf8DeathTest.*:JniUtf8Test.*:JniPaimonUtf8DeathTest.*:JniPaimonUtf8Test.*:JniPaimonWriteBackendTest.*:PaimonWriteBackendFactoryTest.*:PaimonWriteBackendTest.*:PaimonTableSinkOperatorTest.*:PaimonTableWriterTest.*'
-j 6`: **20 passed, 0 failed, 0 skipped**. This includes 10 real-JVM
subprocess tests, 3 JNI allocation-failure tests, and 7 existing Paimon backend
tests. The local test source list was temporarily narrowed; production
libraries and ASAN configuration were retained, and the CMake selection was
restored afterward. Full BE/regression suites, complete Paimon writes, and the
customer-specific JDBC environment were not run.
Style/static checks: clang-format 16, build hygiene, and clang-tidy checks
passed.
- Behavior changed:
- [ ] No.
- [x] Yes. JNI text boundaries preserve standard UTF-8; malformed input
is rejected instead of being silently corrupted. Existing modified UTF-8 APIs
retain their contract.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
