MaxGekk opened a new pull request, #56518:
URL: https://github.com/apache/spark/pull/56518
### What changes were proposed in this pull request?
This PR makes the date field extraction functions (`year`, `quarter`,
`month`, `day`/`dayofmonth`, `dayofyear`, `dayofweek`, `weekday`, `weekofyear`,
`yearofweek`) and the transitive `EXTRACT` / `date_part` date components work
on the nanosecond-precision timestamp types `TIMESTAMP_NTZ(p)` /
`TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`) in ANSI mode.
- Add `AnyTimestampNanoType` (`AbstractDataType`) and
`AnyTimestampNanoTypeExpression` (expression extractor) matching
`TimestampLTZNanosType` / `TimestampNTZNanosType`, mirroring the existing micro
abstractions `AnyTimestampType` / `AnyTimestampTypeExpression`.
- Extend `AnsiGetDateFieldOperationsTypeCoercion` to also match
nanos-timestamp children of `GetDateField` and cast them to `DATE`, exactly as
it already does for micro timestamps. This deliberately does not widen
`AnyTimestampType` (also used as an "accept-as-is, no cast" `inputTypes` gate
on micro-only expressions), and keeps `Cast.canANSIStoreAssign` /
`Cast.canUpCast` strict for `DATE` <-> nanos (the cast is inserted explicitly
by the field-extraction rule).
- Add nanosecond examples to the date field functions'
`@ExpressionDescription`.
No `EXTRACT`-specific change is needed: `EXTRACT(field FROM source)` is a
`RuntimeReplaceable` that rewrites via `DatePart.parseExtractField` to the same
`GetDateField` expressions, so once the `GetDateField` coercion is fixed,
`extract(year from nanos_ts)` and `date_part('year', nanos_ts)` work
transitively in both ANSI and non-ANSI modes. Time-of-day fields
(`HOUR`/`MINUTE`/`SECOND`) were already handled by SPARK-57340.
This is a sub-task of
[SPARK-56822](https://issues.apache.org/jira/browse/SPARK-56822) (SPIP:
Timestamps with nanosecond precision) and builds on
[SPARK-57323](https://issues.apache.org/jira/browse/SPARK-57323) (DATE <->
nanos casts).
### Why are the changes needed?
In ANSI mode (the default since Spark 4.0) the date field functions fail
analysis with `DATATYPE_MISMATCH` on nanosecond timestamps. For example:
```sql
SET spark.sql.timestampNanosTypes.enabled=true;
SELECT year(TIMESTAMP_NTZ '2020-01-01 12:30:15.123456789'::timestamp_ntz(9));
```
fails, because the generic ANSI implicit-cast rule defers to
`Cast.canANSIStoreAssign` (which returns false for nanos -> DATE by design) and
the dedicated `AnsiGetDateFieldOperationsTypeCoercion` rule matched only the
microsecond timestamp types. In non-ANSI mode the functions already work via
the blanket `(_: DatetimeType, _: DatetimeType)` implicit-cast arm.
### Does this PR introduce _any_ user-facing change?
Yes. With `spark.sql.timestampNanosTypes.enabled=true`, date field functions
and the `EXTRACT` / `date_part` date components now work on `TIMESTAMP_NTZ(p)`
/ `TIMESTAMP_LTZ(p)` in ANSI mode. Previously they failed analysis with
`DATATYPE_MISMATCH`; they already worked in non-ANSI mode. The nanosecond
timestamp types are a preview feature (disabled by default), so there is no
change for released behavior.
### How was this patch tested?
- New unit tests in `TimestampNanosFunctionsSuiteBase` (run with ANSI mode
on and off), covering the function form, the `EXTRACT` / `date_part` form, and
the `functions.*` Column API, over a spread of values: leap day, ISO-week and
quarter boundaries, pre-epoch dates, varied precisions (7/8/9) and fractions,
LTZ time-zone date shifts, and NULLs.
- New golden-file queries appended to `timestamp-ltz-nanos.sql` /
`timestamp-ntz-nanos.sql` with regenerated `.sql.out` files.
- `ExpressionInfoSuite` validates the new `@ExpressionDescription` examples.
- `./dev/scalastyle` and scalafmt pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
