peterxcli opened a new issue, #5188:
URL: https://github.com/apache/datafusion-comet/issues/5188
### Describe the bug
When an explicit Parquet read schema requests an ANSI interval type that
does not match the file column's logical type, Comet's native scan differs from
Spark in two directions:
- A Parquet `DATE` column requested as `INTERVAL YEAR TO MONTH` succeeds in
Spark but fails in Comet.
- A Parquet `INT` column requested as `INTERVAL DAY TO SECOND` fails in
Spark but succeeds in Comet, reinterpreting the integers as microseconds.
The second case is a correctness gap because Comet returns rows for a read
that Spark rejects with `PARQUET_COLUMN_DATA_TYPE_MISMATCH`.
### Steps to reproduce
Write a one-column Parquet file, then read it with an explicit schema. Run
each read once with Comet disabled for the Spark reference and once with the
native Parquet scan enabled.
```scala
spark.conf.set("spark.sql.sources.useV1SourceList", "parquet")
spark.sql("SELECT CAST(DATE '2020-01-01' AS DATE) AS c")
.write.parquet(datePath)
spark.read.schema("c INTERVAL YEAR TO MONTH")
.parquet(datePath).collect()
spark.sql("SELECT CAST(1 AS INT) AS c")
.write.parquet(intPath)
spark.read.schema("c INTERVAL DAY TO SECOND")
.parquet(intPath).collect()
```
The full observed matrix on Spark 4.1 is:
| File column | Requested schema | Spark | Comet |
| --- | --- | --- | --- |
| `date` | `interval year to month` | succeeds (raw INT32 value interpreted
as total months) | fails: `expected Interval(YearMonth) but found Date32` |
| `int` | `interval day to second` | `PARQUET_COLUMN_DATA_TYPE_MISMATCH` |
succeeds; values are reinterpreted as microseconds |
| `timestamp` | `interval day to second` | error | error |
| `long` | `interval year to month` | error | error |
### Expected behavior
Comet should match Spark's `ParquetVectorUpdaterFactory` compatibility rules:
- Accept INT32, including `DATE`, for `YearMonthIntervalType` and interpret
the raw value as total months.
- Accept INT64 for `DayTimeIntervalType`.
- Reject INT32/`INT` for `DayTimeIntervalType` with Spark-compatible
schema-mismatch behavior.
- Continue rejecting the timestamp-to-day-time and long-to-year-month
control cases.
### Additional context
The compatibility decision belongs in the native Parquet schema adapter.
Conversion then reaches `parquet_convert_array`, whose Arrow `can_cast_types`
path accepts numeric-to-`Duration`, while its fallback can leave `Date32`
unconverted for an `Interval(YearMonth)` target.
- Schema adapter:
https://github.com/apache/datafusion-comet/blob/2963b0871436766fd14eb15fbe08f57ad50af30e/native/core/src/parquet/schema_adapter.rs#L773-L829
- Array conversion:
https://github.com/apache/datafusion-comet/blob/2963b0871436766fd14eb15fbe08f57ad50af30e/native/core/src/parquet/parquet_support.rs#L166-L260
- PR: https://github.com/apache/datafusion-comet/pull/5161
- Review finding:
https://github.com/apache/datafusion-comet/pull/5161#discussion_r3690464239
- Related generic schema-mismatch work: #3720 and #4297
This issue is limited to schema-on-read compatibility for ANSI interval
targets. Matching-schema interval scans are covered by #5060 / #5161. Add
focused coverage beside the existing primitive schema-conversion cases in
`ParquetReadSuite`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
