Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22341#discussion_r216405553
--- Diff:
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -646,7 +647,47 @@ private[spark] class AppStatusListener(
}
override def onUnpersistRDD(event: SparkListenerUnpersistRDD): Unit = {
- liveRDDs.remove(event.rddId)
+ liveRDDs.remove(event.rddId).foreach { liveRDD =>
+ val executorsToUpdate = new HashSet[LiveExecutor]()
--- End diff --
I'd just call `maybeUpdate` directly when updating the executor. It's
cheaper than inserting into a set, since a duplicate call will just compare the
timestamp of the last update and then do nothing.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
