Re: Determine global watermark via StreamingQueryProgress eventTime watermark String

2021-03-16 Thread Jungtaek Lim
There was a similar question (but another approach) and I've explained the current status a bit. https://lists.apache.org/thread.html/r89a61a10df71ccac132ce5d50b8fe405635753db7fa2aeb79f82fb77%40%3Cuser.spark.apache.org%3E I guess this would also answer your question as well. At least for now, Spa

Determine global watermark via StreamingQueryProgress eventTime watermark String

2021-03-16 Thread dwichman
Hi Spark Developers, Is it possible to reliably determine the current global watermark that is being used in a streaming query via StreamingQueryProgress.onQueryProgress eventTime watermark String? https://spark.apache.org/docs/2.2.1/api/java/org/apache/spark/sql/streaming/StreamingQueryProgress.