rich7420 opened a new issue, #5741: URL: https://github.com/apache/datafusion-comet/issues/5741
### Describe the bug With Comet enabled on Spark 4.1.3, reading a Parquet VARIANT-annotated field as an ordinary `struct<value binary, metadata binary>` does not raise the error Spark requires when `SQLConf.PARQUET_IGNORE_VARIANT_ANNOTATION` is false. Found while restoring the Variant shredding suites for #5569. The existing Spark test `ParquetVariantShreddingSuite / variant logical type annotation - ignore variant annotation` fails with: ```text Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown (ParquetVariantShreddingSuite.scala:277) ``` [CI log](https://github.com/rich7420/datafusion-comet/actions/runs/34022223542/job/101462446982). This is an observed missing exception, not evidence of silent-null corruption in this case. ### Steps to reproduce Use Spark v4.1.3 with the Comet patch, removing the `IgnoreCometSuite` mixin from `ParquetVariantShreddingSuite`. The tested change is [fork PR rich7420/datafusion-comet#3](https://github.com/rich7420/datafusion-comet/pull/3), head `00fa408852c5fa1714fdd00b3de8212c2f2394ce`, based on Comet `7e1984399eb887cd13109698ee55cf2ce150f849`. Run the suite with Comet enabled through the Spark SQL test harness. A focused command after the documented Comet build/install and Spark patch setup is: ```sh NOLINT_ON_COMPILE=true ENABLE_COMET=true ENABLE_COMET_ONHEAP=true \ build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetVariantShreddingSuite' ``` The CI reproduction used the full `sql_core-1` test selection, not this focused command. Environment: Spark 4.1.3, JDK 17, Linux. The [existing test](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVariantShreddingSuite.scala#L233-L290) writes annotated Variant values at the top level and inside structs, arrays and maps. It checks both values of `PARQUET_IGNORE_VARIANT_ANNOTATION`. The `true` arm succeeds; the `false` arm reaches `struct_result.collect()` without throwing. ### Expected behavior Match Spark's reader policy: reject the incompatible struct read when Variant annotations must be respected, and preserve the successful struct read when ignoring annotations is explicitly enabled. Spark expects a `SparkException` caused by an `AnalysisException` with condition `_LEGACY_ERROR_TEMP_3071` and an `Invalid Spark read type` message. ### Additional context Source tracing points to a different boundary from #4084's shredded-Variant fallback: - Spark's [ParquetToSparkSchemaConverter](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L393-L410) validates the physical VARIANT annotation against the requested Spark type. - Comet's [scan schema check](https://github.com/apache/datafusion-comet/blob/7e1984399eb887cd13109698ee55cf2ce150f849/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala#L1067-L1116) examines the requested schema. The manually supplied ordinary struct does not carry the per-field Variant metadata used by `isVariantStruct`, so that fallback does not cover this case. The failure is confirmed by CI; the routing explanation above is source-derived, without a separately captured execution plan. A fix should validate the file annotation or fall back appropriately, without classifying ordinary structs as Variant solely by their child names. The other 14 tests in the two restored suites passed. Track this case separately so #5569 can restore that coverage without retaining a whole-suite exclusion. Related: #5438. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
