Hi, I haven't spent much time on it, but the following DEBUG message from WatermarkTracker sparked my interest :)
I ran a streaming aggregation in Append mode and got the messages: 19/10/08 10:48:56 DEBUG WatermarkTracker: Observed event time stats 0: EventTimeStats(15000,1000,8000.0,2) 19/10/08 10:48:56 INFO WatermarkTracker: Updating event-time watermark from 0 to 5000 ms 19/10/08 10:48:56 DEBUG WatermarkTracker: Event time watermark didn't move: 5000 < 5000 I think the DEBUG message "Event time watermark didn't move" seems incorrect given that the query has just started and "Observed event time stats". It's true that the event-time watermark didn't move if it was 5000 before, but it was not as it has just started from scratch (no checkpointed state). Can anyone shed some light on this? I'll be digging deeper in a bit, but am hoping to get some more info before. Thanks! Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski The Internals of Spark SQL https://bit.ly/spark-sql-internals The Internals of Spark Structured Streaming https://bit.ly/spark-structured-streaming The Internals of Apache Kafka https://bit.ly/apache-kafka-internals Follow me at https://twitter.com/jaceklaskowski
