yadavay-amzn opened a new pull request, #56944: URL: https://github.com/apache/spark/pull/56944
### What changes were proposed in this pull request? Allows nanosecond-precision timestamp columns (`TimestampNTZNanosType`, `TimestampLTZNanosType`) to be used as the event-time column for Structured Streaming watermarks. `withWatermark`'s analyzer type-check now accepts these types, and `EventTimeWatermarkExec` converts the nanosecond event-time value to milliseconds for watermark tracking on the correct scale (`epochMicros` -> `microsToMillis`), matching the existing microsecond-timestamp path. The watermark eviction literal is constructed with the correct nanosecond type. ### Why are the changes needed? Part of nanosecond-precision timestamp support (SPARK-56822). Streaming queries using a nanosecond-precision timestamp as the event-time column previously failed the `withWatermark` type-check (`EVENT_TIME_IS_NOT_ON_TIMESTAMP_TYPE`). ### Does this PR introduce _any_ user-facing change? Yes - `withWatermark` now accepts nanosecond-precision timestamp event-time columns; the watermark advances identically to an equivalent microsecond-timestamp column. ### How was this patch tested? New `EventTimeWatermarkSuite` tests for LTZ-nanos and NTZ-nanos event-time columns asserting correct watermark advancement, plus a microsecond/nanosecond equivalence test. All `EventTimeWatermarkSuite` tests pass. ### Limitations / follow-ups This PR covers the watermark itself (event-time column acceptance + advancement). `window` / `session_window` / `window_time` over nanosecond timestamps and full nanosecond support in streaming stateful operators (e.g. `dropDuplicatesWithinWatermark`) are tracked as follow-ups (SPARK-57829, SPARK-57843); a `TODO(SPARK-57843)` marks the `dropDuplicatesWithinWatermark` path. ### Was this patch authored or co-authored using generative AI tooling? Authored with assistance by Claude Opus 4.8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
