Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5473#discussion_r29105467
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingJobProgressListener.scala
---
@@ -40,6 +43,8 @@ private[streaming] class
StreamingJobProgressListener(ssc: StreamingContext)
private var totalProcessedRecords = 0L
private val receiverInfos = new HashMap[Int, ReceiverInfo]
+ private val batchTimeToBatchUIData = new HashMap[Time, BatchUIData]
--- End diff --
> So we should convert all (waiting/running/completed)BatchInfos to
***BatchUIData. Then we dont need to add this extra batchTimeToBatchUIData any
more.
It's a bit complex here. Because `onJobStart` and `onBatchXXX` messages are
processed in different threads, we may not be able to get the corresponding
BatchInfo in `onJobStart`. In such case, if we convert all
`(waiting/running/completed)BatchInfos` to `***BatchUIData`, we still need to
have an extra data structure to store `BatchUIData`s that does not have a
corresponding `BatchInfo`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
