patrickswedish opened a new pull request, #26401:
URL: https://github.com/apache/datafusion/pull/26401
## Which issue does this PR close?
- Closes #26336
## Rationale for this change
In Spark SQL, `date_part('SECOND', ts)` returns seconds with fractional
microsecond precision as `DECIMAL(8, 6)`. Previously, DataFusion Spark
delegated `date_part('SECOND', ...)` to DataFusion's core `date_part` function,
which casts extracted seconds to `Int32` or `Int64`, completely dropping
fractional seconds.
Furthermore, in SQL logic tests and Spark expressions where the date part
argument is a string literal cast (such as `'SECOND'::string`, parsed as
`Cast(Literal(Utf8("SECOND")), Utf8View)`), `scalar_argument_for_field` in
`datafusion-expr` only checked `Expr::Literal`. Consequently, during logical
plan construction before expression simplification, `scalar_arguments[0]` was
evaluated as `None`, defaulting `SparkDatePart` return type to `Int32`. After
`SimplifyExpressions` reduced the cast to a literal, `scalar_arguments[0]`
became `Some("SECOND")`, deriving `Decimal128(8, 6)` and triggering an
`optimize_projections` schema mismatch assertion.
## What changes are included in this PR?
- `datafusion/spark/src/function/datetime/date_part.rs`:
- `is_second`: Matches `second`, `s`, `sec`, `seconds`, `secs`
(case-insensitive).
- `spark_date_part_second`: Extracts seconds from timestamps (Microsecond,
Millisecond, Second, Nanosecond) and `Date32`/`Date64` directly into
`Decimal128Array(8, 6)` using microsecond integer arithmetic.
- `return_field_from_args`: Returns `DataType::Decimal128(8, 6)` when the
date part argument is a second alias, retaining `DataType::Int32` otherwise.
- `invoke_with_args`: Dispatches to `spark_date_part_second` when second
aliases are evaluated, maintaining Spark compatibility for both scalar and
array columnar inputs.
- `simplify`: Preserves second expressions as
`ExprSimplifyResult::Original` so physical execution handles fractional decimal
extraction.
- Unit tests covering timestamps with microsecond fractions, whole
seconds, null propagation, Date32 types, and return type derivations across all
second aliases.
- `datafusion/expr/src/expr_schema.rs`:
- `scalar_argument_for_field`: Added support for identity and
string-to-string literal casts (`sv.data_type() == field.data_type() ||
(sv.data_type().is_string() && field.data_type().is_string())`). This ensures
that string literal casts like `'SECOND'::string` (parsed as
`Cast(Literal(Utf8), Utf8View)`) correctly resolve their scalar value during
initial plan construction, preventing schema derivation drift between
unoptimized and optimized plans while preserving the test invariant for
non-literal expression casts.
- `datafusion/sqllogictest/test_files/spark/datetime/date_part.slt`:
- Updated expected outputs for `SECOND`, `S`, `SEC`, `SECONDS`, `SECS` to
`query R` returning `45.000000`.
- Added test cases for microsecond timestamp fractions (`45.123456`),
`extract(SECOND FROM ts)`, and null timestamp inputs.
## What is the testing strategy for this PR?
- Unit tests:
- `cargo test -p datafusion-spark --lib` (304 tests passed, 0 failed)
- `cargo test -p datafusion-expr --lib` (284 tests passed, 0 failed)
- Lints/Format:
- `rustfmt --check` (clean, 0 diff)
- `cargo clippy -p datafusion-spark --lib -- -D warnings` (passed, 0
warnings)
- `cargo clippy -p datafusion-expr --lib -- -D warnings` (passed, 0
warnings)
- Integration tests:
- `cargo test -p datafusion-sqllogictest --test sqllogictests --
spark/datetime/date_part` (100% passed)
## Are there any user-facing changes?
Yes. In the Spark compatibility module, `date_part('SECOND', ts)` now
returns `Decimal128(8, 6)` with full fractional precision, matching Apache
Spark behavior.
---
*Diff minimization and pre-flight validation performed with [ZeroSlop™
CI](https://github.com/marketplace/actions/zeroslop-by-ribbsaeter-systems) —
[Ribbsaeter Systems](https://ribbsaetersystems.com/).*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
