antonireus opened a new issue, #12047:
URL: https://github.com/apache/arrow-rs/issues/12047
### Is your feature request related to a problem or challenge?
The Parquet reader fails the whole read when a STRING column holds bytes
that are not valid UTF-8:
`Parquet error: encountered non UTF-8 data: invalid utf-8 sequence of 1
bytes from index 80822`
Files like this are unfortunately common in practice, for example Latin-1
text written as `STRING` by older producers. Other readers handle them:
- Arrow C++ (and so pyarrow and R) and arrow-go don't fail. They put the raw
bytes into the string array and leave the UTF-8 check to later steps
(ValidateFull). Users can still get at the data and clean it up.
- The JVM side doesn't fail either. parquet-java and iceberg-java turn the
bytes into strings with U+FFFD for invalid sequences, and Spark reads them
without error.
- arrow-rs fails, and there is no workaround:
- The pass-through approach of C++ and Go is not possible: StringArray
must hold valid UTF-8, which is why #11013 was closed.
- Reading the column as Binary is rejected (requested Binary but found
Utf8), because validation is keyed on the Parquet UTF8 annotation, not on the
requested Arrow type.
- Stripping the annotation reads the bytes unchecked, which is #11140.
- The error doesn't name the column.
This blocks the Rust engines that need to read what the JVM ecosystem reads.
DataFusion Comet is stuck on exactly this: its native scan fails where Spark
succeeds (apache/datafusion-comet#4121, tracked in
apache/datafusion-comet#4764).
### Describe the solution you'd like
A reader option that picks what happens on invalid UTF-8:
```
pub enum InvalidUtf8 {
/// Fail the read (current behavior, default)
Error,
/// Replace invalid sequences with U+FFFD per value, as
String::from_utf8_lossy does
Replace,
}
ArrowReaderOptions::with_invalid_utf8(InvalidUtf8)
```
- Output is always valid UTF-8
- New behaviour is opt-in, so by default nothing changes
- The replacement is consistent with what the `std` Rust library already
does in `String::from_utf8_lossy`
- Results match the Java/Spark ecosystem, which already replaces, and this
unblocks Comet.
- No cost on valid data. Validation stays as it is. Only the existing error
branch changes.
### Describe alternatives you've considered
- Pass the bytes through, as C++ and Go do: that would be #6701, already
discarded in #11013.
- Read as Binary and convert in the caller: the hint is rejected for STRING
columns. Even if it were allowed, the column loses its string type, so string
predicates and statistics pruning stop working.
- Strip the UTF8 annotation from the metadata, then read as Utf8: the bytes
are not validated and end up in a StringArray (#11140). Once #11144 lands, this
fails exactly as today.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
