andygrove commented on issue #27068: URL: https://github.com/apache/datafusion/issues/27068#issuecomment-6619408618
I tried the DataFusion-side fix described above: never coercing binary file fields to string types, so the column is read as binary (or `BinaryView` for a `Utf8View` table) and the cast to the table's string type validates it. It fixes the bug, but it loses statistics pruning for `binary_as_string` tables, so I'm not proposing it. With a three-row-group file read with `binary_as_string`: | Query | Before | Binary not coerced | | --- | --- | --- | | `WHERE b = 'bbb'` | 1 of 3 row groups matched | 3 of 3 matched | | `WHERE b LIKE 'b%'` | 1 of 3 matched | 3 of 3 matched | Once the column is read as binary, the predicate becomes `CAST(b AS Utf8View) = 'bbb'`, and the cast is not moved to the literal for binary columns. That could be added for `=` and `IN` (`b = X'626262'`), but not for `LIKE`. So I think the better fix is in the Parquet reader: validate UTF-8 whenever the target type is a string, so the column can still be read directly as a string and pruning keeps working. That is being discussed in apache/arrow-rs#12047, together with an opt-in `InvalidUtf8::Replace` policy for `binary_as_string` users whose data is not valid UTF-8. DataFusion would then expose that option for Parquet scans. The nested case (#27065) is different: #27067 only restores the 55.2 behavior there, so it costs no pruning that 55.2 had. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
