andygrove commented on issue #4764: URL: https://github.com/apache/datafusion-comet/issues/4764#issuecomment-6618986238
Status and proposed direction for the remaining scan gaps, as of 2026-11-03. **Gap A** (#4121, `UTF8`-annotated STRING columns): unchanged. The native scan still errors where Spark succeeds. **Gap C** (BINARY columns read as STRING): - Top-level values reach a `Utf8` array unvalidated (#7032), and so do map keys (#7360). - Nested values decode to U+FFFD on `main` since #7106. On DataFusion 56 they are unvalidated again (#7469), because apache/datafusion#25193 made DataFusion's file-schema coercion read nested binary fields as strings directly. That is reported as apache/datafusion#27065, with a fix in draft apache/datafusion#27067 that restores the 55.2 behavior. #7469 tracks it for the DataFusion 56 upgrade (#6410). **Proposed direction: one reader option for both gaps.** apache/arrow-rs#12047 proposes an opt-in `ArrowReaderOptions::with_invalid_utf8(InvalidUtf8::Replace)` that replaces invalid UTF-8 with U+FFFD instead of failing. That is option 1 from the Gap A notes above, but in the reader itself, so the column keeps its string type and string pruning still works. I commented there (https://github.com/apache/arrow-rs/issues/12047#issuecomment-6618965660) asking for two things: - The check should apply whenever the target type is a string, not only to `UTF8`-annotated columns. Without that, `Replace` would not cover Gap C. - `Replace` should match the JVM's decoder, or be pluggable. `from_utf8_lossy` emits a different number of U+FFFD than the JVM for some sequences, such as encoded surrogates. If it lands that way, the end state for Comet is: - DataFusion exposes the option, and Comet's native scan sets `Replace`. - #4121, #7032, #7360 and #7469 close. `hll.sql` comes back, and the nested decode from #7106 is no longer needed. - The remaining difference from Spark is the documented byte-level round trip, plus any replacement-count differences if arrow-rs only offers `from_utf8_lossy` semantics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
