Hi Spark Developers, Is it possible to reliably determine the current global watermark that is being used in a streaming query via StreamingQueryProgress.onQueryProgress eventTime watermark String?
https://spark.apache.org/docs/2.2.1/api/java/org/apache/spark/sql/streaming/StreamingQueryProgress.html The intention would be to precede the query watermark function with something like a map function and compare event times with the assumed global watermark to determine if the event will be dropped (i.e. too late). If StreamingQueryProgress.onQueryProgress eventTime watermark does not accurately reflect the current global watermark, is there another way to reliably determine it? Thanks for your help. -Derek -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
