andygrove commented on issue #4199: URL: https://github.com/apache/datafusion-comet/issues/4199#issuecomment-6218852488
This no longer reproduces on `main`. apache/arrow-rs#9855 lets parquet-rs accept the `UNKNOWN` logical type on any physical type, and the parquet 59 crate Comet uses includes it. I checked at c230bf4e98 on Spark 4.1.3. A read of a Spark-written file with a `NullType` column runs in the native scan when the query prunes that column, and falls back to Spark when the query reads it, and both match Spark. `issue #4136: struct with only NullType fields in file (SPARK-54220)` in `CometNativeReaderSuite` also passes with its `assume(false)` removed. #7052 drops the Spark 4.1 known-limitation entry for this from `spark-versions.md`. Two skips still point here and can go in a follow-up: that `assume(false)`, and the `IgnoreComet` on the SPARK-54220 test in `dev/diffs/4.1.3.diff` and `dev/diffs/4.2.0.diff`. Closing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
