[
https://issues.apache.org/jira/browse/SPARK-30448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves resolved SPARK-30448.
-----------------------------------
Fix Version/s: 3.0.0
Resolution: Fixed
> accelerator aware scheduling enforce cores as limiting resource
> ---------------------------------------------------------------
>
> Key: SPARK-30448
> URL: https://issues.apache.org/jira/browse/SPARK-30448
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Major
> Fix For: 3.0.0
>
>
> For the first version of accelerator aware scheduling(SPARK-27495), the SPIP
> had a condition that we can support dynamic allocation because we were going
> to have a strict requirement that we don't waste any resources. This means
> that the number of number of slots each executor has could be calculated from
> the number of cores and task cpus just as is done today.
> Somewhere along the line of development we relaxed that and only warn when we
> are wasting resources. This breaks the dynamic allocation logic if the
> limiting resource is no longer the cores. This means we will request less
> executors then we really need to run everything.
> We have to enforce that cores is always the limiting resource so we should
> throw if its not.
> I guess we could only make this a requirement with dynamic allocation on, but
> to make the behavior consistent I would say we just require it across the
> board.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
