ump45nose opened a new pull request, #7089: URL: https://github.com/apache/datafusion-comet/pull/7089
## Which issue does this PR close? Part of #7034. ## Rationale for this change Spark has no vectorized-reader updater that reads an INT64 `TIMESTAMP(NANOS)` column as `timestamp` or `timestamp_ntz` (it only reads it as `bigint` under `spark.sql.legacy.parquet.nanosAsLong`), so it fails with `PARQUET_COLUMN_DATA_TYPE_MISMATCH`. The native scan instead cast the values to microseconds and returned them. ## What changes are included in this PR? `spark_has_updater` now rejects a nanosecond Arrow timestamp read as a microsecond one. INT96 is coerced to microseconds before this check, so at the top level and inside structs and maps a nanosecond timestamp can only come from `TIMESTAMP(NANOS)`. The rejection is deferred to decode time like the other `getUpdater` rejections (`RejectOnNonEmpty`), and reading the column as `bigint` is still allowed. Below a list there is one catch: an INT96 leaf in a legacy list layout is not coerced today (#7029, apache/datafusion#26393) and shows up as a timezone-free `Timestamp(Nanosecond)`, which looks the same as `TIMESTAMP(NANOS, isAdjustedToUTC=false)`. Rejecting that would break legacy INT96 arrays that read correctly now, so below a list only the UTC-adjusted form is rejected. To know whether a leaf is below a list, `check_conversion` now delegates to `check_nested_conversion`, which carries an `in_list` flag down to `spark_has_updater`. Once #7029 is fixed upstream the exemption can go and #7034 can be closed. I updated the known-issues entry to describe the case that remains. ## How are these changes tested? - `parquet_nanos_timestamp_read_as_timestamp_errors` writes a `TIMESTAMP(NANOS)` column, with and without a timezone, and checks that reading it as `timestamp` and `timestamp_ntz` fails with the Spark-shaped error (`Column: [[t]]`, `Found: INT64`, `Expected: timestamp`). - `nanos_timestamp_rejection_below_a_list` checks the verdicts from `check_conversion`: rejected in structs, in maps and for UTC-adjusted values in lists; still accepted for timezone-free values in lists and for reads as `bigint`. Both tests fail without the change. I ran the `datafusion-comet` crate's lib tests and `cargo clippy --all-targets -- -D warnings` on it. I did not run the Spark/JVM suites. AI disclosure: the change, the tests and this description were written with an AI coding assistant. The part most worth a careful look is the list exemption and whether a timezone-free nanosecond leaf below a list can come from anything other than these two sources. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
