andygrove opened a new pull request, #329:
URL: https://github.com/apache/datafusion-spark/pull/329
Closes #149.
## Summary
Adds a Spark-compatible `make_date(year, month, day)` to `datafusion-spark`,
ported from DataFusion Comet's `datetime_funcs/make_date.rs`.
- Arguments are coerced to INT (integer, floating-point, decimal and string
inputs), the same way `date_from_unix_date` does it. BOOLEAN is rejected.
- If any input is NULL, the result is NULL. Rows with a NULL input are
skipped before any check, so under ANSI an invalid month next to a NULL year
does not raise an error.
- Invalid dates (month 13, day 0, February 29 in a non-leap year, a year
outside ±999999999) return NULL when ANSI mode is off. With
`datafusion.execution.enable_ansi_mode = true` they raise
`[DATETIME_FIELD_OUT_OF_BOUNDS.WITH_SUGGESTION]` with Spark's message and
SQLSTATE 22023. The messages are the ones `java.time.LocalDate.of` produces,
and the checks run in the same order.
- The full `java.time` year range is supported, including year 0 and
negative years. Day counts are computed in `i64`.
- A valid date whose day count does not fit in DATE raises
`[ARITHMETIC_OVERFLOW]` in both modes. This matches Spark, where
`MathUtils.toIntExact` is outside the `DateTimeException` handler.
- With all-scalar arguments and zero rows, an empty array comes back without
the arguments being checked. As a result, `SELECT make_date(2024, 13, 1) FROM t
WHERE false` under ANSI returns no rows, as it does in Spark.
- The ANSI mode is read from `ScalarFunctionArgs::config_options`, as `abs`
and `elt` do. `apache/main` has no crate-root `EvalMode` yet.
- The result field is always nullable, because ANSI mode is not known at
planning time.
- Spark has no `try_make_date`, so none is added.
## Differences from Comet's version
- **Out-of-range epoch day:** Comet returns NULL (ANSI off) or
`DATETIME_FIELD_OUT_OF_BOUNDS` (ANSI on) for a valid date outside the DATE
range, for example `make_date(5881580, 7, 12)`. Spark 4.2.0 raises
`ARITHMETIC_OVERFLOW` in both modes, and this version does the same.
- **Mode configuration:** Comet takes `fail_on_error` in the constructor.
This version reads `enable_ansi_mode` from the session config.
- **Signature:** Comet uses `Signature::any(3)` and casts every argument to
Int32 at runtime. This version uses a coercible INT signature, so a BOOLEAN
argument is rejected at planning time, as in Spark.
- **Empty batches:** Comet's `MakeDateExpr` wrapper skips the whole
expression on empty batches, including errors raised while evaluating the
arguments (for example an ANSI cast of `'bad'`). This version only skips the
date check for all-scalar arguments when `number_rows == 0`. Errors raised
while evaluating the arguments of a row-less batch are a DataFusion planner
concern and are out of scope here.
- **Errors:** Errors are DataFusion execution errors carrying Spark's
message text rather than Comet's `SparkError` type.
## Known divergences from Spark (tracked by #88)
These are the same cast-related gaps as in `date_from_unix_date`. They are
captured as commented-out `.slt` queries:
- With ANSI off, Spark returns NULL for `make_date('abc', '2', '3')`.
DataFusion raises a cast error.
- With ANSI off, Spark wraps an out-of-range BIGINT to INT, which gives an
invalid year and so NULL, for example `make_date(3000000000, 1, 1)`. DataFusion
raises a cast error.
## Testing
- **Spark source:** Read Spark's `MakeDate` in 3.5.8 and 4.1.1. Semantics
are the same in both; only the name of the error helper differs.
- **PySpark 4.2.0:** Checked every expected value against PySpark 4.2.0 with
ANSI both on and off. That includes valid and invalid dates, every error
message, the year 0, negative and large year cases, the overflow boundaries,
NULL handling, array inputs and type coercion.
- **SLT runner:** `crates/sqllogictest/scripts/pyspark_slt_runner.py` on
`datetime/make_date.slt` gave 34 pass and 1 skipped (`arrow_typeof` does not
exist in Spark).
- **Unit tests:** Ported Comet's unit tests and added tests for the ANSI
messages, the overflow error, scalar and array invocation, and the empty-batch
behavior.
- **Commands that passed:**
- `cargo fmt --all`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `cargo test -p datafusion-spark` (346 passed)
- `cargo test -p datafusion-spark-sqllogictest` (249 of 249 files passed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]