You Zhou created SPARK-58309:
--------------------------------
Summary: now() and current_timestamp() are not anchored to the
batch timestamp in streaming queries
Key: SPARK-58309
URL: https://issues.apache.org/jira/browse/SPARK-58309
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: You Zhou
In micro-batch streaming, CURRENT_LIKE temporal expressions are rewritten to
CurrentBatchTimestamp so their value is anchored to the batch timestamp
(persisted in the offset log) and is reproducible across batch
replays/restarts. The rewrite in MicroBatchExecution only matched
CurrentTimestamp, missing now() (Now, a sibling under CurrentTimestampLike) and
current_time() (CurrentTime, which also needs a TimeType branch in
CurrentBatchTimestamp.toLiteral). As a result these fold to wall-clock time at
plan-optimization time and produce non-deterministic values on replay.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
