andygrove opened a new pull request, #329:
URL: https://github.com/apache/datafusion-spark/pull/329

   Closes #149.
   
   ## Summary
   
   Adds a Spark-compatible `make_date(year, month, day)` to `datafusion-spark`, 
ported from DataFusion Comet's `datetime_funcs/make_date.rs`.
   
   - Arguments are coerced to INT (integer, floating-point, decimal and string 
inputs), the same way `date_from_unix_date` does it. BOOLEAN is rejected.
   - If any input is NULL, the result is NULL. Rows with a NULL input are 
skipped before any check, so under ANSI an invalid month next to a NULL year 
does not raise an error.
   - Invalid dates (month 13, day 0, February 29 in a non-leap year, a year 
outside ±999999999) return NULL when ANSI mode is off. With 
`datafusion.execution.enable_ansi_mode = true` they raise 
`[DATETIME_FIELD_OUT_OF_BOUNDS.WITH_SUGGESTION]` with Spark's message and 
SQLSTATE 22023. The messages are the ones `java.time.LocalDate.of` produces, 
and the checks run in the same order.
   - The full `java.time` year range is supported, including year 0 and 
negative years. Day counts are computed in `i64`.
   - A valid date whose day count does not fit in DATE raises 
`[ARITHMETIC_OVERFLOW]` in both modes. This matches Spark, where 
`MathUtils.toIntExact` is outside the `DateTimeException` handler.
   - With all-scalar arguments and zero rows, an empty array comes back without 
the arguments being checked. As a result, `SELECT make_date(2024, 13, 1) FROM t 
WHERE false` under ANSI returns no rows, as it does in Spark.
   - The ANSI mode is read from `ScalarFunctionArgs::config_options`, as `abs` 
and `elt` do. `apache/main` has no crate-root `EvalMode` yet.
   - The result field is always nullable, because ANSI mode is not known at 
planning time.
   - Spark has no `try_make_date`, so none is added.
   
   ## Differences from Comet's version
   
   - **Out-of-range epoch day:** Comet returns NULL (ANSI off) or 
`DATETIME_FIELD_OUT_OF_BOUNDS` (ANSI on) for a valid date outside the DATE 
range, for example `make_date(5881580, 7, 12)`. Spark 4.2.0 raises 
`ARITHMETIC_OVERFLOW` in both modes, and this version does the same.
   - **Mode configuration:** Comet takes `fail_on_error` in the constructor. 
This version reads `enable_ansi_mode` from the session config.
   - **Signature:** Comet uses `Signature::any(3)` and casts every argument to 
Int32 at runtime. This version uses a coercible INT signature, so a BOOLEAN 
argument is rejected at planning time, as in Spark.
   - **Empty batches:** Comet's `MakeDateExpr` wrapper skips the whole 
expression on empty batches, including errors raised while evaluating the 
arguments (for example an ANSI cast of `'bad'`). This version only skips the 
date check for all-scalar arguments when `number_rows == 0`. Errors raised 
while evaluating the arguments of a row-less batch are a DataFusion planner 
concern and are out of scope here.
   - **Errors:** Errors are DataFusion execution errors carrying Spark's 
message text rather than Comet's `SparkError` type.
   
   ## Known divergences from Spark (tracked by #88)
   
   These are the same cast-related gaps as in `date_from_unix_date`. They are 
captured as commented-out `.slt` queries:
   
   - With ANSI off, Spark returns NULL for `make_date('abc', '2', '3')`. 
DataFusion raises a cast error.
   - With ANSI off, Spark wraps an out-of-range BIGINT to INT, which gives an 
invalid year and so NULL, for example `make_date(3000000000, 1, 1)`. DataFusion 
raises a cast error.
   
   ## Testing
   
   - **Spark source:** Read Spark's `MakeDate` in 3.5.8 and 4.1.1. Semantics 
are the same in both; only the name of the error helper differs.
   - **PySpark 4.2.0:** Checked every expected value against PySpark 4.2.0 with 
ANSI both on and off. That includes valid and invalid dates, every error 
message, the year 0, negative and large year cases, the overflow boundaries, 
NULL handling, array inputs and type coercion.
   - **SLT runner:** `crates/sqllogictest/scripts/pyspark_slt_runner.py` on 
`datetime/make_date.slt` gave 34 pass and 1 skipped (`arrow_typeof` does not 
exist in Spark).
   - **Unit tests:** Ported Comet's unit tests and added tests for the ANSI 
messages, the overflow error, scalar and array invocation, and the empty-batch 
behavior.
   - **Commands that passed:**
     - `cargo fmt --all`
     - `cargo clippy --all-targets --all-features -- -D warnings`
     - `cargo test -p datafusion-spark` (346 passed)
     - `cargo test -p datafusion-spark-sqllogictest` (249 of 249 files passed)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to