[
https://issues.apache.org/jira/browse/SPARK-30749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-30749:
----------------------------------
Summary: stage level scheduling: Better cleanup of Resource profiles (was:
Better cleanup of Resource profiles)
> stage level scheduling: Better cleanup of Resource profiles
> -----------------------------------------------------------
>
> Key: SPARK-30749
> URL: https://issues.apache.org/jira/browse/SPARK-30749
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: Thomas Graves
> Priority: Major
>
> In the initial stage level scheduling jiras we aren't doing a great job of
> cleaning up datastructures when the ResourceProfile is done being used. This
> is mostly because its hard to tell when its done being used.
> We should find a way to clean up better.
> Also in the dynamic allocation manager, if the resource profile is not being
> used anymore we should be more active about getting rid of the executors that
> are up. Especially if there is a minimum number that is set, we can kill
> those off.
>
> I think this can be done as followup to the main feature
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
