raminqaf opened a new pull request, #29690:
URL: https://github.com/apache/flink/pull/29690
## What is the purpose of the change
A cast from VARIANT accepts a timestamp only for its own type. A variant
`TIMESTAMP_LTZ` reaches `TIMESTAMP_LTZ(p)` and fails for every other datetime
target, so a user has to know the stored kind and nest two casts. With this PR
a stored timestamp reaches the other datetime types the way a regular cast from
its SQL type does, using the session time zone.
With `table.local-time-zone` set to `Europe/Berlin` and `v_timestamp_ltz`
holding the instant `2024-01-01 23:30:00.123456Z`:
| Query | Before
| After |
|-----------------------------------------|----------------------------------------------------------------------------|---------------------------|
| `CAST(v_timestamp_ltz AS TIMESTAMP(3))` | fails `Cannot cast a VARIANT
TIMESTAMP_LTZ value to TIMESTAMP(3). ...` | `2024-01-02 00:30:00.123` |
| `CAST(v_timestamp_ltz AS DATE)` | fails `VariantTypeException:
Expected type DATE but got TIMESTAMP_LTZ` | `2024-01-02` |
| `CAST(v_timestamp_ltz AS TIME(3))` | fails `Cannot cast a VARIANT
TIMESTAMP_LTZ value to TIME(3). ...` | `00:30:00.123` |
| Stored kind | Before | Adds
|
|-------------------------------------|--------------------|---------------------------------------|
| `TIMESTAMP`, `TIMESTAMP_NS` | `TIMESTAMP(p)` |
`TIMESTAMP_LTZ(p)`, `DATE`, `TIME(p)` |
| `TIMESTAMP_LTZ`, `TIMESTAMP_LTZ_NS` | `TIMESTAMP_LTZ(p)` | `TIMESTAMP(p)`,
`DATE`, `TIME(p)` |
| `DATE` | `DATE` | nothing
|
| `TIME` | `TIME(p)` | nothing
|
Decisions worth a look:
- **DATE and TIME stay own-type only.** Widening them would add a part the
value never had: a time of day, a date, or a zone offset. The regular TIME to
TIMESTAMP cast shows the problem, since it fills in 1970-01-01 (FLINK-24423).
- **DATE and TIME targets drop part of a timestamp,** while the numeric rule
rejects `7.2` to `INT`. Cutting off fractional seconds beyond the target
precision is already allowed, and Spark, Snowflake and Trino apply their
regular cast rules here.
- **The session time zone is the one regular casts use.** The cast rule
passes `context.getSessionZoneId()` and `VARIANT_GET` passes
`table.local-time-zone`, both at planning time, the same zone the regular cast
rules put into the generated code.
- **Values are read through java.time.** The regular TIMESTAMP to DATE and
TIME casts divide epoch milliseconds and are wrong before 1970 (FLINK-41280).
`DateTimeUtils.truncate` has the same problem for a precision of 3 or less, so
before this PR a stored `1969-12-31 23:59:58.5` cast to `TIMESTAMP(0)` gave
`23:59:59`. The VARIANT path now truncates the nano-of-second and gives
`23:59:58`. The regular cast still has that bug, which is out of scope here.
- **A DATE target that gets another kind fails with the standard VARIANT
cast message** instead of the raw `VariantTypeException`.
## Brief change log
- `VariantCastUtils`: `toTimestamp`, `toTimestampLtz` and `toTime` accept
the other timestamp kinds and take the session time zone, and the new `toDate`
does the same for `DATE`
- `FromVariantConverter` passes the session `ZoneId` to the datetime
readers
- Docs: the VARIANT cast table and a new bullet in `data-types.md`, en and
zh
## Verifying this change
This change added tests and can be verified as follows:
- `CastRulesTest`: the four entries that asserted a TIMESTAMP and a
TIMESTAMP_LTZ never cross now cast, in UTC and CET, for the micro and
nanosecond kinds. The `DATE` and `TIME(3)` specs gain timestamp sources,
including a date before 1970, and `TIMESTAMP(0)` gains a truncation case before
1970. The `DATE`, `TIMESTAMP` and `TIMESTAMP_LTZ` specs check that a `TIME` or
`DATE` value still fails, the `DATE` one with the standard message.
- `FromVariantConverterTest#testConverterSurvivesJavaSerialization` also
covers the session zone that a datetime reader captures.
- No new ITCase. The `TIMESTAMP_LTZ` to `STRING` cases in
`CastFunctionITCase` and `VariantFunctionsITCase` already check that the cast
and `VARIANT_GET` hand the session time zone to the converter. Both classes
pass unchanged.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes, the
VARIANT to datetime cast readers. A same-kind cast does the same work as
before, and an error message is only built when a cast fails.
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code 2.1.314 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
