andygrove opened a new pull request, #27067: URL: https://github.com/apache/datafusion/pull/27067
## Which issue does this PR close? - Closes #27065. ## Rationale for this change Reading a Parquet file with a nested BINARY field (a list element or a struct field) through a table schema that declares that field as a string produces a string array that was never checked for valid UTF-8. When the bytes are not valid UTF-8, the query panics in a debug build. In a release build it returns an invalid `StringArray`, which is unsound for anything that reads the values as `&str`. On 55.2.0 the same query fails with `Encountered non UTF-8 data`. #25193 made the file-schema coercion recurse into nested types so that the view coercions (`Utf8` to `Utf8View`, `Binary` to `BinaryView`) reach nested fields (#25192). The binary-to-string coercion came along with it. That coercion hands the Parquet reader a string type, and the reader only validates UTF-8 for columns annotated `UTF8`, which a BINARY column is not. #25342 already kept map keys and values out of it for this reason. ## What changes are included in this PR? - `coerce_data_type` applies the binary-to-string coercion to top-level fields only, as before #25193. Struct fields and list-like children are coerced with it turned off, so nested binary fields stay binary in the file schema and are converted by the validating cast. The view coercions still apply at every level. - `coerce_child` loses its `binary_to_string` parameter, since every caller now passes `false`. - The module docs describe the rule for nested fields. The top-level case is unchanged: a top-level binary column read as a string still goes through the reader without validation, as it has for a long time. Fixing that needs the Parquet reader to validate whenever the target type is a string, which is apache/arrow-rs#12063. ## What is the testing strategy for this PR? - `nested_coercion_preserves_leaf_conversion_rules` now checks that nested view coercions still apply and that nested binary-to-string coercions do not. - The new `nested_coercion_leaves_binary_list_and_struct_children_to_the_validating_cast` writes a file with invalid UTF-8 in a binary list element and a binary struct field, checks that a string table schema leaves both binary, and checks that the reader returns valid binary arrays with the original bytes. It follows the existing map test, `nested_coercion_leaves_binary_map_children_to_the_validating_cast`. ## Are there any user-facing changes? A query that reads invalid UTF-8 from a nested binary field as a string returns an error instead of panicking or producing invalid data, which restores the 55.2.0 behavior. Valid data reads as before, through the cast instead of the reader. No public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
