[
https://issues.apache.org/jira/browse/SPARK-44134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735746#comment-17735746
]
Thomas Graves commented on SPARK-44134:
---------------------------------------
I'm working on a fix for this
> Can't set resources (GPU/FPGA) to 0 when they are set to positive value in
> spark-defaults.conf
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-44134
> URL: https://issues.apache.org/jira/browse/SPARK-44134
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.2.0
> Reporter: Thomas Graves
> Priority: Major
>
> With resource aware scheduling, if you specify a default value in the
> spark-defaults.conf, a user can't override that to set it to 0.
> Meaning spark-defaults.conf has something like:
> {{spark.executor.resource.\{resourceName}.amount=1}}
> {{spark.task.resource.\{resourceName}.amount}} =1
> {{}}
> If the user tries to override when submitting an application with
> {{{}spark.executor.resource.\{resourceName}.amount{}}}=0 and
> {{{}{}}}{{{}spark.task.resource.\{resourceName}.amount{}}}{{ =0, it gives the
> user an error:}}
> {{}}
> {code:java}
> 23/06/21 09:12:57 ERROR Main: Failed to initialize Spark session.
> org.apache.spark.SparkException: No executor resource configs were not
> specified for the following task configs: gpu
> at
> org.apache.spark.resource.ResourceProfile.calculateTasksAndLimitingResource(ResourceProfile.scala:206)
> at
> org.apache.spark.resource.ResourceProfile.$anonfun$limitingResource$1(ResourceProfile.scala:139)
> at scala.Option.getOrElse(Option.scala:189)
> at
> org.apache.spark.resource.ResourceProfile.limitingResource(ResourceProfile.scala:138)
> at
> org.apache.spark.resource.ResourceProfileManager.addResourceProfile(ResourceProfileManager.scala:95)
> at
> org.apache.spark.resource.ResourceProfileManager.<init>(ResourceProfileManager.scala:49)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:455)
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704)
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953){code}
> This used to work, my guess is this may have gotten broken with the stage
> level scheduling feature.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
