sam-1112 opened a new pull request, #5956: URL: https://github.com/apache/datafusion-comet/pull/5956
## Which issue does this PR close? Part of #5103 (temporal truncation only). ## Rationale for this change #5103 asked to stop maintaining a parallel truncation kernel where DataFusion already has a compatible physical `date_trunc`. This PR wires the compatible scalar-format `trunc(date, fmt)` and `date_trunc(fmt, timestamp)` paths to `datafusion-functions` 55.1.0, and keeps Comet's per-row format, dictionary unwrap, and session-timezone pre-shift. This is not a full replacement of Comet's truncation kernel. DataFusion's coarse `date_trunc` converts through nanoseconds, so Spark-legal values outside the TimestampNanosecond range (approximately 1678–2262, for example year 3333) still use the existing calendar fallback. ## What changes are included in this PR? - Normalize Spark format aliases (`YEAR`/`YYYY`/`YY`, `QUARTER`, `MONTH`/`MON`/`MM`, Monday-based `WEEK`), then call DataFusion's physical `date_trunc`. - Date32 uses a Date32 → Timestamp → `date_trunc` → Date32 sandwich when the value fits the upstream range. - Timestamp scalar-format truncation uses the same physical `date_trunc` after the existing session-timezone pre-shift. - Values outside DataFusion's TimestampNanosecond coarse-trunc range fall back to the existing Comet calendar kernel. - Per-row format arrays, dictionary inputs, and session-timezone handling are unchanged. - Non-UTC native `date_trunc` remains `Incompatible` (#2649). DST SQL coverage uses `allowIncompatible=true`; shuffle may still insert the existing #4515 schema-align cast. ## How are these changes tested? Spark compatibility and DST-boundary tests: - SQL: aliases, leap years, pre-epoch dates, year-3333 fallback, dictionary encoding, and NULL format (`trunc_date.sql`, `trunc_timestamp.sql`). - DST gap/overlap: `America/Los_Angeles`, `America/New_York`, `America/Sao_Paulo`. - Scala fuzz in `CometTemporalExpressionSuite` for 2024 (upstream path) and 3333 (fallback path). - Rust kernel tests for scalar/date/timestamp, dictionary, wide-range fallback, and DST gap/overlap. Local runs: ```bash cd native cargo fmt --all -- --check cargo test -p datafusion-comet-spark-expr kernels::temporal::tests cargo test -p datafusion-comet-spark-expr cd .. make core ./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometSqlFileTestSuite trunc" ./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometTemporalExpressionSuite" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
