andygrove opened a new pull request, #327:
URL: https://github.com/apache/datafusion-spark/pull/327

   Closes #151.
   
   ## Summary
   
   Adds a Spark-compatible `dayofweek(date)`, which returns an INT from 1 
(Sunday) to 7 (Saturday). It is ported from Comet's 
`datetime_funcs/day_of_week.rs`.
   
   - New `crates/spark/src/function/datetime/dayofweek.rs`. The result comes 
straight from the `Date32` epoch day: `((days + 4) mod 7) + 1`, computed in 
`i64`. This matches Spark's 
`LocalDate.ofEpochDay(days).getDayOfWeek.plus(1).getValue`, which is defined 
for every `Int`.
   - `invoke_with_args` handles scalar and array inputs, NULL and untyped NULL, 
and dictionary-encoded `Date32`. For a dictionary, it maps the values, rewraps 
the keys, and unpacks the result to plain `Int32`.
   - The signature is the same `date_signature()` that `dayname` and 
`monthname` use. Strings and timestamps are implicitly cast to DATE, as in 
Spark.
   - Documented with `#[user_doc]` and registered in `datetime/mod.rs`.
   - `weekday` now uses the same epoch-day helper instead of arrow's 
`date_part(DayOfWeekMonday0)`. arrow builds a chrono date for each row, so 
dates outside chrono's range (such as `date_from_unix_date(±2^31)`) came back 
NULL. Spark returns 1 and 4 for those dates, and so does `weekday` now. The 
`weekday` signature is unchanged.
   
   ## Differences from Comet's version
   
   - Comet uses `Signature::any(1)` and expects the planner to hand it a 
`Date32`. Here the function has a coercible date signature, so strings and 
timestamps work from SQL. Comet's name `spark_dayofweek` becomes `dayofweek`.
   - A scalar input returns a scalar here. Comet always expands it to an array.
   - `return_field_from_args` passes the input's nullability through to the 
result, the same as `weekday`.
   - The kernel, the null-density switch between `unary` and `unary_opt`, and 
the dictionary handling are the same as in Comet. I found no behavioral 
difference between Comet and Spark 4.2.0 for `Date32` input.
   
   Known differences from Spark, which also apply to the existing `dayname` and 
`weekday` and are left as commented-out cases in the `.slt`:
   - With ANSI mode off, Spark returns NULL for a malformed string. 
DataFusion's cast errors regardless of ANSI mode (#88).
   - Spark casts a timestamp with a time zone to a date in the session time 
zone. Here it is cast in the timestamp's own time zone (#88).
   
   ## Testing
   
   - Every expected value was checked against PySpark 4.2.0. This covers all 
seven weekdays, the epoch, a leap day, 0001-01-01, 9999-12-31, the 1582 
Gregorian cutover, `date_from_unix_date(-2147483648)` = 3 and `(2147483647)` = 
6, NULL and untyped NULL (`int`), strings, TIMESTAMP_NTZ, TIMESTAMP in UTC and 
in America/New_York, malformed strings with ANSI on and off, and an INT 
argument (DATATYPE_MISMATCH).
   - `crates/sqllogictest/scripts/pyspark_slt_runner.py` on `dayofweek.slt` and 
`weekday.slt`: every `dayofweek` record passes on Spark 4.2.0. Skipped records 
are the `arrow_typeof` checks and the dictionary case, marked `skipif spark`. 
The one `weekday` mismatch, `weekday('not-a-date')`, was already there (strings 
are not cast for `weekday`, #41).
   - Unit tests ported from Comet: the week around the epoch, nulls at both 
densities, the full `i32` range, agreement with `weekday` over a 400-year 
cycle, dictionary input, non-date input rejected, and scalar input. A 
nullability test was added.
   - Passing: `cargo fmt --all`, `cargo clippy --all-targets --all-features -- 
-D warnings`, `cargo test -p datafusion-spark` and `cargo test -p 
datafusion-spark-sqllogictest` (249 of 249 files).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to