viirya commented on a change in pull request #30427:
URL: https://github.com/apache/spark/pull/30427#discussion_r528895988
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala
##########
@@ -126,6 +126,58 @@ private[ui] class StreamingQueryStatisticsPage(parent:
StreamingQueryTab)
<br />
}
+ def generateWatermark(
+ query: StreamingQueryUIData,
+ minBatchTime: Long,
+ maxBatchTime: Long,
+ jsCollector: JsCollector): Seq[Node] = {
+ // This is made sure on caller side but put it here to be defensive
+ require(query.lastProgress != null)
+ if (query.lastProgress.eventTime.containsKey("watermark")) {
+ val watermarkData = query.recentProgress.flatMap { p =>
+ val batchTimestamp = parseProgressTimestamp(p.timestamp)
+ val watermarkValue =
parseProgressTimestamp(p.eventTime.get("watermark"))
+ if (watermarkValue > 0L) {
+ // seconds
+ Some((batchTimestamp, ((batchTimestamp - watermarkValue) / 1000.0)))
Review comment:
Thanks @xuanyuanking for raising this discussion.
> OK I took too many efforts to write the comment and wrote too late. In
short, if we can pick the second metric to compare with global watermark, it
should be min instead of max. If Spark also picks min event time to construct
watermark, we should pick max to see how much the output is lagging due to slow
watermark advance.
I agree. Actually my first thought is to use min event time instead of batch
time in this graph.
I think a ideal approach should be able to select different base time for
constructing this graph, e.g. min event time or batch time. I am not sure if
current UI component supports this kind of feature. But for current change, I
think it should be good enough for use cases except for event time is far from
clock time. That is why I gave +1 for this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
