PorridgeSwim opened a new pull request, #57480: URL: https://github.com/apache/spark/pull/57480
### What changes were proposed in this pull request? In `MicroBatchExecution`, the per-batch rewrite that replaces `CURRENT_LIKE` temporal expressions with `CurrentBatchTimestamp` only matched `CurrentTimestamp`, `CurrentDate` and `LocalTimestamp`. Two expressions carrying the `CURRENT_LIKE` tree pattern were missed: - `Now` (`now()`) -- a sibling of `CurrentTimestamp` under the `CurrentTimestampLike` base class. The match on the leaf type `CurrentTimestamp` did not catch it. - `CurrentTime` (`current_time()` / `localtime()`, the `TIME` value type). This PR: - Widens the `CurrentTimestamp` case to `CurrentTimestampLike`, covering both `current_timestamp()` and `now()`. - Adds a `CurrentTime` case, and a `TimeType` branch to `CurrentBatchTimestamp.toLiteral` that derives the time-of-day (nanoseconds since midnight, in the session time zone) from the batch timestamp, truncated to the type's precision. (Because the batch timestamp is millisecond-resolution, sub-millisecond digits of `current_time()` are zero in streaming.) ### Why are the changes needed? Expressions left unrewritten survive to the per-batch optimizer, where `ComputeCurrentTime` folds them using the wall clock at plan-optimization time. That value is not reproducible when a batch is replayed after failure or on restart from a checkpoint, breaking the determinism guarantee that `CurrentBatchTimestamp` (anchored to the batch timestamp persisted in the offset log) exists to provide. `now()` and `current_time()` were therefore non-deterministic across batch replays. ### Does this PR introduce _any_ user-facing change? Yes. In a streaming query, `now()` and `current_time()` now return the batch timestamp (stable across replay/restart of the same batch), consistent with `current_timestamp()`, `current_date()` and `localtimestamp()`, instead of the wall clock at plan-optimization time. ### How was this patch tested? Added a parameterized test in `StreamSuite` covering `current_timestamp()`, `now()`, `current_date()` and `current_time()`. Using a manual clock it asserts the exact per-batch value, then purges the last batch's commit and restarts with the wall clock advanced far ahead, asserting the replayed batch reproduces its persisted timestamp rather than the advanced clock. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
