[
https://issues.apache.org/jira/browse/SPARK-39617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexandru Tofan updated SPARK-39617:
------------------------------------
Description:
When updating Spark 3.0.2 to Spark 3.2.1 I get the following error:
{code:java}
Exception in thread "main" org.apache.spark.SparkException: Driver cores must
be a positive number
at
org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:634)
at
org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:257)
at
org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:234)
at
org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:119)
at
org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1026)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1026)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code}
Obviously I set the driver-cores property but it doesn't matter, it is not
used. I'm suspecting that it is something about the Mesos Dispatcher, it
receives the property because it is being displayed correctly in the
Dispatcher's UI, but it is not forwarded correctly to Spark-Submit, and
therefore fails.
Update:
It happens when submitting a *Spark 3.2.1* job with *Mesos* in {*}Cluster
mode{*}.
was:
When updating Spark 3.0.2 to Spark 3.2.1 I get the following error:
{code:java}
Exception in thread "main" org.apache.spark.SparkException: Driver cores must
be a positive number
at
org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:634)
at
org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:257)
at
org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:234)
at
org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:119)
at
org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1026)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1026)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code}
Obviously I set the driver-cores property but it doesn't matter, it is not
used. I'm suspecting that it is something about the Mesos Dispatcher, it
receives the property because it is being displayed correctly in the
Dispatcher's UI, but it is not forwarded correctly to Spark-Submit, and
therefore fails.
> Driver cores must be a positive number
> --------------------------------------
>
> Key: SPARK-39617
> URL: https://issues.apache.org/jira/browse/SPARK-39617
> Project: Spark
> Issue Type: Bug
> Components: Mesos
> Affects Versions: 3.2.1
> Environment: Spark 3.2.1
> Mesos
> Java 8 OpenJDK
> Reporter: Alexandru Tofan
> Priority: Major
> Labels: mesos, spark-submit
>
> When updating Spark 3.0.2 to Spark 3.2.1 I get the following error:
> {code:java}
> Exception in thread "main" org.apache.spark.SparkException: Driver cores must
> be a positive number
> at
> org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:634)
> at
> org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:257)
> at
> org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:234)
> at
> org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:119)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1026)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1026)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code}
> Obviously I set the driver-cores property but it doesn't matter, it is not
> used. I'm suspecting that it is something about the Mesos Dispatcher, it
> receives the property because it is being displayed correctly in the
> Dispatcher's UI, but it is not forwarded correctly to Spark-Submit, and
> therefore fails.
>
> Update:
> It happens when submitting a *Spark 3.2.1* job with *Mesos* in {*}Cluster
> mode{*}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
