linzch3 opened a new pull request, #12066:
URL: https://github.com/apache/arrow-rs/pull/12066
# Which issue does this PR close?
- Closes #11579.
# Rationale for this change
`RecordDecoder::flush` validates UTF-8 once over the concatenated field
bytes. When invalid bytes on either side of a delimiter or line terminator join
into a valid sequence (e.g. `0xCD` | `0x9C`), validation passes, and
`StringRecord::get` then hands out `&str`s that are not valid UTF-8 through
`get_unchecked`.
# What changes are included in this PR?
After the `from_utf8` check, `flush` also checks that every field offset is
a char boundary of the validated data, and returns the existing "Encountered
invalid UTF-8 data for line {line} and field {field}" error for the field that
ends mid-character. The error construction moves into a closure so both paths
share it. The extra check is O(1) per field.
AI disclosure: the change and tests were prepared with an AI coding
assistant (Grok Bot). The tests were run locally with and without the fix.
# Are these changes tested?
Yes. Two cases were added to `test_invalid_utf8`: a split sequence across a
field delimiter and one across a line terminator. Without the fix, the reader
returns a batch and the test panics while formatting the invalid string.
# Are there any significant user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
