[
https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083959#comment-17083959
]
Hongze Zhang commented on SPARK-31437:
--------------------------------------
Thanks [~tgraves] and I too found some related discussion around here[1].
I am glad to take this on but now I am still in the phase of understanding the
concept and codes. Go back to the topic, So far one of my questions is, what's
the point of binding task resource requirements to the executor instances? Can
we break the consolidated ResourceProfile up into something like executor's
ResourceProfile and task's ResourceRequest? If like that, we can remove the
strict id requirement then issue like this one would be pretty easy to deal
with. At the same time we can still have the opportunity to keep the overall
logic simple: we can choose one strategy from several to create ResourceProfile
from incoming ResourceRequest. The strictest strategy is exactly the same as
what we are doing currently, and we can use more looser strategies for cases
like mine.
[1] https://github.com/apache/spark/pull/26284#issuecomment-553481259
> Try assigning tasks to existing executors by which required resources in
> ResourceProfile are satisfied
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-31437
> URL: https://issues.apache.org/jira/browse/SPARK-31437
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 3.1.0
> Reporter: Hongze Zhang
> Priority: Major
>
> By the change in [PR|https://github.com/apache/spark/pull/27773] of
> SPARK-29154, submitted tasks are scheduled onto executors only if resource
> profile IDs strictly match. As a result Spark always starts new executors for
> customized ResourceProfiles.
> This limitation makes working with process-local jobs unfriendly. E.g. Task
> cores has been increased from 1 to 4 in a new stage, and executor has 8
> slots, it is expected that 2 new tasks can be run on the existing executor
> but Spark starts new executors for new ResourceProfile. The behavior is
> unnecessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
