mikhail-melnik opened a new pull request, #3576: URL: https://github.com/apache/parquet-java/pull/3576
<!-- Thanks for opening a pull request! If you're new to Parquet-Java, information on how to contribute can be found here: https://parquet.apache.org/docs/contribution-guidelines/contributing Please open a GitHub issue for this pull request: https://github.com/apache/parquet-java/issues/new/choose and format pull request title as below: GH-${GITHUB_ISSUE_ID}: ${SUMMARY} or simply use the title below if it is a minor issue: MINOR: ${SUMMARY} --> ### Rationale for this change `VariantUtil.getString()`, `getDictKey()`, and `getMetadataMap()` used `new String(byte[], ...)` without a charset, relying on the JVM platform default. On JDK <= 17 with `LC_ALL=C` (the build invocation recommended in the README), the platform default becomes `US-ASCII`, causing multi-byte UTF-8 sequences in Variant string values and metadata keys to be decoded as multiple characters instead of one. The [Variant binary encoding spec](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) mandates UTF-8 for all string data. The write path already uses `StandardCharsets.UTF_8` explicitly, this PR aligns the read path to match. ### What changes are included in this PR? `VariantUtil.java`: add `StandardCharsets.UTF_8` to six `new String(byte[], ...)` call sites across `getString()`, `getDictKey()`, and `getMetadataMap()`. ### Are these changes tested? Yes, the existing `testParseUnicodeString` test in `TestVariantParseJson` covers this. It was already written to catch this exact case but only failed when running under `LC_ALL=C` (i.e. on JDK <= 17 with the README-recommended build invocation). With this fix, `LC_ALL=C ./mvnw test -pl parquet-variant` passes cleanly. ### Are there any user-facing changes? No API changes. Behavior change only for JVMs running with a non-UTF-8 default charset, where string values containing non-ASCII characters were previously silently corrupted on read. <!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the actual Github issue id. --> Closes #3573 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
