Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19751#discussion_r156711461
--- Diff:
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -772,4 +813,118 @@ private[spark] class AppStatusListener(
}
}
+ private def cleanupExecutors(count: Long): Unit = {
+ // Because the limit is on the number of *dead* executors, we need to
calculate whether
+ // there are actually enough dead executors to be deleted.
+ val threshold = conf.get(MAX_RETAINED_DEAD_EXECUTORS)
+ val dead = count - activeExecutorCount
--- End diff --
KVStore has this:
```java
/**
* Returns the number of items of the given type which match the given
indexed value.
*/
long count(Class<?> type, String index, Object indexedValue) throws
Exception;
```
so with an api change you could get the right number directly from the
store. (though this conflicts with my other comment about not using
`kvstore.count()` at all in the trigger, which I think is more important.)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
