ump45nose opened a new pull request, #7227: URL: https://github.com/apache/datafusion-comet/pull/7227
## Which issue does this PR close? Closes #7214. ## Rationale for this change Spark's vectorized reader has no updater for these reads and fails with `PARQUET_COLUMN_DATA_TYPE_MISMATCH`, but the native scan casts the values instead: - an INT96 or INT64 timestamp column read as `string`, `byte`/`short`/`int`, `float`/`double` or `decimal` (returns text, raw numbers or NULL); - an INT96 column read as `bigint` (returns microseconds since the epoch). An INT64 timestamp read as `bigint` is fine in Spark and stays allowed. ## What changes are included in this PR? - `spark_has_updater` rejects a timestamp read as a string/binary, a narrower integer, a float or a decimal. These are deferred like the other `getUpdater` rejections (`RejectOnNonEmpty`). - INT96 and INT64 timestamps have the same Arrow type by the time the check runs, so the INT96 `bigint` rule needs the Parquet type. Like `TIMESTAMP(NANOS)` in #7089, the footer check (`mark_parquet_leaves`) now marks INT96 leaves with a private `INT96_TIMESTAMP_KEY`, and files with INT96 leaves take the deferred check path even without row filters. To carry both markers, the `nanos_timestamp: bool` parameter became a small `ParquetLeaf` enum (`Other`, `NanosTimestamp`, `Int96`). ## How are these changes tested? - `parquet_timestamp_read_as_non_timestamp_errors`: a `TIMESTAMP(MICROS)` column, with and without a timezone, read as `string`, `int`, `double` and `decimal(20,0)` fails with the Spark-shaped error; read as `bigint` it still works. - `int96_read_as_bigint_errors`: an INT96 file read as `bigint` fails, with row filters on and off. - `int96_bigint_rejection_keys_on_the_parquet_type`: a marked INT96 leaf is rejected as `bigint` and still accepted as `timestamp`; an unmarked timestamp leaf is still accepted as `bigint`. All three fail without the change. I ran the `parquet::` lib tests in the `datafusion-comet` crate (331 passed) and `cargo clippy -p datafusion-comet --all-targets -- -D warnings`. In the full lib run, only the native UDF tests failed, because I hadn't built `comet-test-udfs`. I did not run the Spark/JVM suites. AI disclosure: I wrote the change, the tests and this description with help from an AI coding assistant. The part most worth checking is the INT96 marker, which uses the same footer-marking path as `TIMESTAMP(NANOS)`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
