[
https://issues.apache.org/jira/browse/SPARK-24954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578419#comment-16578419
]
Imran Rashid commented on SPARK-24954:
--------------------------------------
I was thinking more about this, and realized you'll also run into issues w/
static allocation, and a multi-tenant cluster with pre-emption. You might keep
gaining resources and having them taken away from you.
I'm not sure if there is anything we can really do about that, though, other
than telling users to not run w/ preemption, though its hard to tell whether
maybe they have things configured in a way they'll never get pre-empted.
> Fail fast on job submit if run a barrier stage with dynamic resource
> allocation enabled
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-24954
> URL: https://issues.apache.org/jira/browse/SPARK-24954
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.4.0
> Reporter: Jiang Xingbo
> Assignee: Jiang Xingbo
> Priority: Blocker
> Fix For: 2.4.0
>
>
> Since we explicitly listed "Support running barrier stage with dynamic
> resource allocation" a Non-Goal in the design doc, we shall fail fast on job
> submit if running a barrier stage with dynamic resource allocation enabled,
> to avoid some confusing behaviors (can refer to SPARK-24942 for some
> examples).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
