peterxcli opened a new pull request, #5735: URL: https://github.com/apache/datafusion-comet/pull/5735
## Which issue does this PR close? Closes #5090. ## Rationale for this change Use Arrow's existing cast kernel for the remaining hand-written temporal conversion in #5090: Date32 to TIMESTAMP_NTZ. The other three locations in the issue were already handled by #5177. ## What changes are included in this PR? - Replace the NTZ builder loop with `cast_with_options` targeting `Timestamp(Microsecond, None)`. - Use `DEFAULT_CAST_OPTIONS` (`safe: false`) so out-of-range dates return an error. Arrow 59 uses checked multiplication here, matching Spark's checked `daysToMicros` conversion instead of the previous loop's release-mode wrapping or debug-mode panic. - Preserve the session-timezone path and Spark-specific DST ambiguity/gap resolution. - Extend the existing NTZ regression test with the positive and negative microsecond boundaries and overflowing dates. ## How are these changes tested? - `cd native && cargo test -p datafusion-comet-spark-expr --lib conversion_funcs::temporal::tests --offline`: 2 tests passed after rebasing onto upstream main. Covers the timezone path, NTZ session-timezone independence, negative dates, nulls, timezone annotation, and overflow boundaries. - `cargo fmt --manifest-path native/Cargo.toml --all -- --check` and `git diff --check upstream/main...HEAD` passed. ### Kernel microbenchmark Standalone release-mode comparison of the previous builder loop against the new Arrow cast, using Arrow 59.3.0 and Rust 1.96.0 on macOS arm64. Release settings: thin LTO, one codegen unit, overflow checks disabled. This isolates the conversion kernel; it does not measure Spark query performance or the full Comet expression path. Inputs contain deterministic dates before and after the epoch, with irregular null placement. Both implementations receive identical arrays, and output equality is checked before timing every case. Input generation is outside the timed region; allocation and output destruction are included. Each case warms up for 200 ms, then collects 21 paired samples with alternating execution order and black-boxed inputs/outputs. Reported values are median time per batch; speedup is the ratio of medians. Null percentages describe the generation thresholds. | Rows | Nulls | Previous loop | Arrow cast | Speedup | |---:|---:|---:|---:|---:| | 1,024 | 0% | 0.904 µs | 0.579 µs | 1.56× | | 1,024 | 10% | 3.551 µs | 0.795 µs | 4.47× | | 1,024 | 50% | 3.789 µs | 0.537 µs | 7.06× | | 1,024 | 100% | 2.496 µs | 0.220 µs | 11.35× | | 8,192 | 0% | 6.477 µs | 3.704 µs | 1.75× | | 8,192 | 10% | 26.945 µs | 5.505 µs | 4.89× | | 8,192 | 50% | 28.996 µs | 3.401 µs | 8.53× | | 8,192 | 100% | 18.628 µs | 0.839 µs | 22.22× | These are local measurements from one run, not CI performance guarantees. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
