Github user rmarsch commented on a diff in the pull request:
https://github.com/apache/spark/pull/7714#discussion_r35685773
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -614,6 +621,22 @@ private[worker] class Worker(
webUi.stop()
metricsSystem.stop()
}
+
+ private def trimFinishedExecutorsIfNecessary(): Unit = {
+ if (finishedExecutors.size > retainedExecutors) {
+ finishedExecutors.take(math.max(finishedExecutors.size / 10,
1)).foreach{
--- End diff --
I was about to work on this same issue until finding this PR already
posted. I had one observation that I wanted to bring up for discussion.
`scala.collection.concurrent.HashMap` does not preserve insertion ordering when
you perform operations on it for iterator or traversal methods. So `take`, for
example may be able to remove recent additions where the user might prefer to
just lose the oldest executors from the list.
[LinkedHashMap](http://www.scala-lang.org/api/2.11.5/index.html#scala.collection.mutable.LinkedHashMap)
does provide the guarantee on insertion ordering being preserved for its
operations. It comes at the cost of more memory overhead to provide this
guarantee, but it may be worth it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
