dwsmith1983 opened a new pull request, #6862: URL: https://github.com/apache/datafusion-comet/pull/6862
## Which issue does this PR close? Closes #6131. ## Rationale for this change With `spark.sql.parquet.fieldId.read.enabled=true`, the native scan matches fields by id on the Arrow schema it gets after DataFusion's INT96 coercion. That coercion rebuilt struct, list and map fields without their metadata, so an id that lived only on a container was lost and the column came back as nulls while Spark read the data. Spark writes timestamps as INT96 by default, so any id-bearing container with a timestamp under it was affected. The DataFusion 55.2.0 upgrade in #6766 resolved this. It carries apache/datafusion#25895 (the branch-55 backport of apache/datafusion#24790), which keeps field metadata when coercing INT96 timestamps. Comet needs no code change, so this PR adds tests that keep the case covered. ## What changes are included in this PR? Tests in `ParquetReadSuite`. Each writes INT96 timestamps with Spark, then reads with field ids on and the id-bearing field renamed, so the id is the only way to match it: - an id on a struct, list or map column holding a timestamp - an id on a struct nested below a struct and below a list, with an id on its timestamp leaf as well - an id on a Variant column next to a timestamp (Spark 4.0+) ## How are these changes tested? The new tests fail on the commit before #6766 (`a56db5a6c`), where Comet returns nulls for each renamed column, and pass on `main` with Spark 3.4, 3.5, 4.0 and 4.1. For the nested case, Spark's own vectorized reader before 4.0 rejects the renamed struct below the list, so that test compares with Spark from 4.0 on and checks the native scan and the expected rows on every version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
