[
https://issues.apache.org/jira/browse/SPARK-37305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-37305.
----------------------------------
Resolution: Invalid
Resolving as Invalid: this is a usage/how-to question rather than a report of a
specific Spark defect or a concrete change proposal. Apache Spark uses JIRA to
track bugs and improvements; usage questions are best asked on the
[email protected] mailing list (https://spark.apache.org/community.html) or
Stack Overflow (tag apache-spark), where a wider audience can help. Please
re-open with a concrete reproducer or a specific proposed code/doc change if
this is actually a bug or an actionable improvement. Thanks!
> Spark Dynamic Resource Allocation in Standalone Mode
> ----------------------------------------------------
>
> Key: SPARK-37305
> URL: https://issues.apache.org/jira/browse/SPARK-37305
> Project: Spark
> Issue Type: Question
> Components: Spark Core
> Affects Versions: 3.1.1
> Reporter: Fernando
> Priority: Major
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Hello,
> I have some questions about whether it is possible to use "Dynamic Resource
> Allocation" in standalone mode (2 workers and 1 master). I have been trying
> the suggestions proposed in your code, either by modifying by adding
> configuration variables to the SparkSession, or by including them directly as
> environment variables in Docker. None of these modifications have worked, and
> until one task is finished, I don't have the resources to do the next one
> (FIFO mode tasks). I wanted to ask if there is a way to run the tasks in
> parallel between several workers in standalone mode, since according to your
> website, it is possible, but it is disabled by default. Following the
> official documentation, I understand that it is possible, but realising the
> changes, I don't see any change in the concurrency:
> _Standalone mode: By default, applications submitted to the standalone mode
> cluster will run in FIFO (first-in-first-out) order, and each application
> will try to use all available nodes. You can limit the number of nodes an
> application uses by setting the spark.cores.max configuration property in it,
> or change the default for applications that don’t set this setting through
> spark.deploy.defaultCores. Finally, in addition to controlling cores, each
> application’s spark.executor.memory setting controls its memory use._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
