andygrove opened a new issue, #7034:
URL: https://github.com/apache/datafusion-comet/issues/7034
### Describe the bug
The native Parquet scan reads an INT64 `TIMESTAMP(NANOS)` column through a
schema that declares `timestamp`, and on Spark 4.x one that declares
`timestamp_ntz`, and returns the values truncated to microseconds. Spark has no
reader for that combination and raises
`[FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH]`. It applies to top-level
and nested columns. pyarrow 13 and later write nanosecond timestamps by
default, so a catalog table declared over pyarrow output reaches this.
`spark_has_updater` (`native/core/src/parquet/schema_adapter.rs` around
lines 553-713) mirrors `ParquetVectorUpdaterFactory.getUpdater` for the
conversions it checks, but has no rule that rejects a nanosecond timestamp read
as a microsecond Spark timestamp, so the adapter casts it. Spark reads
`TIMESTAMP(NANOS)` only as `bigint`, with
`spark.sql.legacy.parquet.nanosAsLong=true`, and Comet matches that case.
### Steps to reproduce
Write a file with parquet-mr's `RecordConsumer` (or pyarrow) holding
`optional int64 t (TIMESTAMP(NANOS,true))` and `optional group s { optional
int64 t (TIMESTAMP(NANOS,true)); }`, with one row whose values are
`1600000000123456789`. Then, with `spark.sql.session.timeZone=UTC`:
```scala
spark.read.schema("t timestamp").parquet(path).collect()
spark.read.schema("s struct<t: timestamp>").parquet(path).collect()
spark.read.schema("t timestamp_ntz").parquet(path).collect()
```
On `main` at 66c63bd307:
| Read schema | Spark 3.5.9 and 4.1.3 | Comet native scan |
| --- | --- | --- |
| `t timestamp` | `PARQUET_COLUMN_DATA_TYPE_MISMATCH ... column: [t],
physicalType: INT64, logicalType: timestamp` | `2020-09-13 12:26:40.123456` UTC
|
| `s struct<t: timestamp>` | the same error for `[s, t]` | `{2020-09-13
12:26:40.123456}` |
| `t timestamp_ntz` | the same error | 4.1.3: `2020-09-13T12:26:40.123456`.
3.5.9: the same error as Spark |
### Expected behavior
The native scan rejects a nanosecond timestamp read as `timestamp` or
`timestamp_ntz` the way Spark does, deferred to decode time like the other
updater rejections (`RejectOnNonEmpty`).
### Additional context
Found in a nested-type audit of the native scans. Comet returns plausible
values where Spark fails, so this is a compatibility gap rather than a wrong
result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
