andygrove opened a new issue, #7360: URL: https://github.com/apache/datafusion-comet/issues/7360
### Describe the bug When a Parquet BINARY column without the UTF8 annotation is read as STRING (through a read schema or `spark.sql.parquet.binaryAsString=true`), #7106 decodes nested values with Spark's lossy UTF-8 decoding, except for map keys. Spark tells map keys apart by their bytes: decoding would turn the keys `41 FF 42` and `41 EF BF BD 42` into the same string, so a lookup of one returns the other's value and `map_from_entries` fails with `DUPLICATED_MAP_KEY`. To avoid that, the map-key arm of `parquet_convert_array_impl` (`native/core/src/parquet/parquet_support.rs`) keeps the raw bytes and wraps them with `StringArray::new_unchecked`. If a key is not valid UTF-8, that breaks the constructor's precondition. With Arrow 59.3.0, `StringArray::try_new` with the same offsets, bytes and validity rejects the key `41 FF 42` at byte index 1. `new_unchecked` requires that the checked constructor would succeed ([contract](https://docs.rs/arrow-array/59.3.0/arrow_array/array/struct.GenericByteArray.html#method.new_unchecked)), so a later string operation on the key can produce an invalid Rust `&str`. The fixture in the #7106 test "native scan decodes invalid UTF-8 in nested BINARY read as string" builds such a key. Its byte-level assertions pass because they don't check that the array is valid. Reported by @sunchao in https://github.com/apache/datafusion-comet/pull/7340#discussion_r4428977680. This is the same unsound state that top-level values already reach (#7032, #4764). It is not a regression: before #7106 these keys became NULL through Arrow's safe cast. Affected: `main`, `branch-1.2` (including 1.2.0), and `branch-1.1` once #7340 merges. ### Steps to reproduce Build the key array from the #7106 fixture with `StringArray::try_new` instead of `new_unchecked`; it returns an invalid UTF-8 error. In a debug build, any operation that validates the array fails the same way. ### Expected behavior The native scan neither admits invalid UTF-8 into a `Utf8` map key array nor merges keys that Spark keeps distinct. Options suggested in the review: 1. Keep the keys as binary until a boundary that preserves the bytes. 2. Fall back to Spark for a scan that reads a BINARY map key as STRING. 3. Validate with the checked constructor and return an error instead of building the array. This is sound but fails queries that Spark answers. ### Additional context The general policy for invalid UTF-8 entering native string arrays is tracked in #4764. A fix there for the top-level path should cover this one too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
