Rich-T-kid commented on code in PR #11144:
URL: https://github.com/apache/arrow-rs/pull/11144#discussion_r4564861682
##########
parquet/src/arrow/record_reader/mod.rs:
##########
@@ -217,6 +222,11 @@ where
self.padding_threshold = Some(threshold);
}
+ /// Validate decoded values as UTF-8 even if the column is not annotated
as a string.
+ pub fn set_validate_utf8(&mut self, validate_utf8: bool) {
Review Comment:
Could we add a two row group test for this? A small reproducer is
`[b"valid", b"\xFF"]` written with `set_max_row_group_row_count(Some(1))`, then
read as `Utf8` with batch size 1. The first `next()` should succeed and the
second should return the UTF-8 error. That exercises the new decoder created by
the second `set_page_reader` call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]