hassaanch23 commented on issue #11140: URL: https://github.com/apache/arrow-rs/issues/11140#issuecomment-5749539496
I'd like to work on this, unless you're already on it. The plan: give the byte-array decoders the target Arrow type, so `validate_utf8` is set whenever the reader is asked to produce `Utf8`, `LargeUtf8` or `Utf8View`, not only when the column carries the `UTF8` annotation. The same decision appears in three places — `byte_array.rs:192`, `byte_view_array.rs:154` and `byte_array_dictionary.rs:243` — so all three need it, including a dictionary whose value type is a string. The three `make_*_reader` functions already receive the target type, so the flag can travel to the decoder through the record reader, the way `padding_threshold` already does. `ColumnValueDecoder` would get a `set_validate_utf8` method with a default no-op, so existing implementations are unaffected. Reading such a file would then fail with an error instead of returning an array that breaks the string invariant. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
