ump45nose commented on PR #7227: URL: https://github.com/apache/datafusion-comet/pull/7227#issuecomment-6895055172
@andygrove thanks for tracing that. I went with the first option in `172e6b10`: the runtime check now rejects `Timestamp(Second, _) -> Timestamp` as well. A second-unit timestamp can only come from an `ARROW:schema` hint over an unannotated INT64 leaf, so it now follows the same `INT64 -> Timestamp` rule the footer check applies after `mark_parquet_leaves` rewrites it to `Int64`. Both paths reject the read, as Spark does. `Second -> bigint` and `Second -> decimal(20,0)` are unchanged, so the earlier `canReadAsDecimal` case still reads. New test `parquet_arrow_writer_timestamp_second_read_as_timestamp_errors` writes `Timestamp(Second, UTC)` with ArrowWriter and reads it as `timestamp` and `timestamp_ntz`, with row filters off and on. All four reads are rejected naming the column. Without the change, the row-filters-off read succeeds. Local: `cargo test -p datafusion-comet --lib -- parquet::` 335 passed (1 ignored), `cargo clippy -p datafusion-comet --all-targets -- -D warnings` clean. The commit also applies `cargo fmt` to two spots from my earlier commit that the current stable rustfmt formats differently. (AI-assisted change and reply.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
