[jira] [Commented] (SPARK-3404) SparkSubmitSuite fails with "spark-submit exits with code 1"
[
https://issues.apache.org/jira/browse/SPARK-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126698#comment-14126698
]
Apache Spark commented on SPARK-3404:
-
User 'srowen' has created a pull request for this issue:
https://github.com/apache/spark/pull/2328
> SparkSubmitSuite fails with "spark-submit exits with code 1"
>
>
> Key: SPARK-3404
> URL: https://issues.apache.org/jira/browse/SPARK-3404
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 1.0.2, 1.1.0
>Reporter: Sean Owen
>Priority: Critical
>
> Maven-based Jenkins builds have been failing for over a month. For example:
> https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-pre-YARN/
> It's SparkSubmitSuite that fails. For example:
> https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-pre-YARN/541/hadoop.version=2.0.0-mr1-cdh4.1.2,label=centos/consoleFull
> {code}
> SparkSubmitSuite
> ...
> - launch simple application with spark-submit *** FAILED ***
> org.apache.spark.SparkException: Process List(./bin/spark-submit, --class,
> org.apache.spark.deploy.SimpleApplicationTest, --name, testApp, --master,
> local, file:/tmp/1409815981504-0/testJar-1409815981505.jar) exited with code 1
> at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:837)
> at
> org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(SparkSubmitSuite.scala:311)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply$mcV$sp(SparkSubmitSuite.scala:291)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply(SparkSubmitSuite.scala:284)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply(SparkSubmitSuite.scala:284)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> ...
> - spark submit includes jars passed in through --jar *** FAILED ***
> org.apache.spark.SparkException: Process List(./bin/spark-submit, --class,
> org.apache.spark.deploy.JarCreationTest, --name, testApp, --master,
> local-cluster[2,1,512], --jars,
> file:/tmp/1409815984960-0/testJar-1409815985029.jar,file:/tmp/1409815985030-0/testJar-1409815985087.jar,
> file:/tmp/1409815984959-0/testJar-1409815984959.jar) exited with code 1
> at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:837)
> at
> org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(SparkSubmitSuite.scala:311)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply$mcV$sp(SparkSubmitSuite.scala:305)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply(SparkSubmitSuite.scala:294)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply(SparkSubmitSuite.scala:294)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> ...
> {code}
> SBT builds don't fail, so it is likely to be due to some difference in how
> the tests are run rather than a problem with test or core project.
> This is related to http://issues.apache.org/jira/browse/SPARK-3330 but the
> cause identified in that JIRA is, at least, not the only cause. (Although, it
> wouldn't hurt to be doubly-sure this is not an issue by changing the Jenkins
> config to invoke {{mvn clean && mvn ... package}} {{mvn ... clean package}}.)
> This JIRA tracks investigation into a different cause. Right now I have some
> further information but not a PR yet.
> Part of the issue is that there is no clue in the log about why
> {{spark-submit}} exited with status 1. See
> https://github.com/apache/spark/pull/2108/files and
> https://issues.apache.org/jira/browse/SPARK-3193 for a change that would at
> least print stdout to the log too.
> The SparkSubmit program exits with 1 when the main class it is supposed to
> run is not found
> (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L322)
> This is for example SimpleApplicationTest
> (https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala#L339)
> The test actually submits an empty JAR not containing this class. It relies
> on {{spark-submit}} finding the class within the compiled test-classes of the
> Spark project. However it does seem to be compiled and
[jira] [Commented] (SPARK-3404) SparkSubmitSuite fails with "spark-submit exits with code 1"
[
https://issues.apache.org/jira/browse/SPARK-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121653#comment-14121653
]
Sean Owen commented on SPARK-3404:
--
It's 100% repeatable in Maven for me locally, which seems to be Jenkins'
experience too. I don't see the same problem with SBT (/dev/run-tests) locally,
although I can't say I run that regularly.
I could rewrite the SparkSubmitSuite to submit a JAR file that actually
contains the class it's trying to invoke. Maybe that's smarter? the problem
here seems to be the vagaries of what the run-time classpath is during an SBT
vs Maven test. Would anyone second that?
Separately it would probably not hurt to get in that change that logs stdout /
stderr from the Utils method.
> SparkSubmitSuite fails with "spark-submit exits with code 1"
>
>
> Key: SPARK-3404
> URL: https://issues.apache.org/jira/browse/SPARK-3404
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 1.0.2, 1.1.0
>Reporter: Sean Owen
>Priority: Critical
>
> Maven-based Jenkins builds have been failing for over a month. For example:
> https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-pre-YARN/
> It's SparkSubmitSuite that fails. For example:
> https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-pre-YARN/541/hadoop.version=2.0.0-mr1-cdh4.1.2,label=centos/consoleFull
> {code}
> SparkSubmitSuite
> ...
> - launch simple application with spark-submit *** FAILED ***
> org.apache.spark.SparkException: Process List(./bin/spark-submit, --class,
> org.apache.spark.deploy.SimpleApplicationTest, --name, testApp, --master,
> local, file:/tmp/1409815981504-0/testJar-1409815981505.jar) exited with code 1
> at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:837)
> at
> org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(SparkSubmitSuite.scala:311)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply$mcV$sp(SparkSubmitSuite.scala:291)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply(SparkSubmitSuite.scala:284)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply(SparkSubmitSuite.scala:284)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> ...
> - spark submit includes jars passed in through --jar *** FAILED ***
> org.apache.spark.SparkException: Process List(./bin/spark-submit, --class,
> org.apache.spark.deploy.JarCreationTest, --name, testApp, --master,
> local-cluster[2,1,512], --jars,
> file:/tmp/1409815984960-0/testJar-1409815985029.jar,file:/tmp/1409815985030-0/testJar-1409815985087.jar,
> file:/tmp/1409815984959-0/testJar-1409815984959.jar) exited with code 1
> at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:837)
> at
> org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(SparkSubmitSuite.scala:311)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply$mcV$sp(SparkSubmitSuite.scala:305)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply(SparkSubmitSuite.scala:294)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply(SparkSubmitSuite.scala:294)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> ...
> {code}
> SBT builds don't fail, so it is likely to be due to some difference in how
> the tests are run rather than a problem with test or core project.
> This is related to http://issues.apache.org/jira/browse/SPARK-3330 but the
> cause identified in that JIRA is, at least, not the only cause. (Although, it
> wouldn't hurt to be doubly-sure this is not an issue by changing the Jenkins
> config to invoke {{mvn clean && mvn ... package}} {{mvn ... clean package}}.)
> This JIRA tracks investigation into a different cause. Right now I have some
> further information but not a PR yet.
> Part of the issue is that there is no clue in the log about why
> {{spark-submit}} exited with status 1. See
> https://github.com/apache/spark/pull/2108/files and
> https://issues.apache.org/jira/browse/SPARK-3193 for a change that would at
> least print stdout to the log too.
> The SparkSubmit program exits with 1 when the main class it is supposed to
> run is not found
> (https://github.com/a
[jira] [Commented] (SPARK-3404) SparkSubmitSuite fails with "spark-submit exits with code 1"
[
https://issues.apache.org/jira/browse/SPARK-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121650#comment-14121650
]
Andrew Or commented on SPARK-3404:
--
I have updated the title to reflect this.
> SparkSubmitSuite fails with "spark-submit exits with code 1"
>
>
> Key: SPARK-3404
> URL: https://issues.apache.org/jira/browse/SPARK-3404
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 1.0.2, 1.1.0
>Reporter: Sean Owen
>Priority: Critical
>
> Maven-based Jenkins builds have been failing for over a month. For example:
> https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-pre-YARN/
> It's SparkSubmitSuite that fails. For example:
> https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-pre-YARN/541/hadoop.version=2.0.0-mr1-cdh4.1.2,label=centos/consoleFull
> {code}
> SparkSubmitSuite
> ...
> - launch simple application with spark-submit *** FAILED ***
> org.apache.spark.SparkException: Process List(./bin/spark-submit, --class,
> org.apache.spark.deploy.SimpleApplicationTest, --name, testApp, --master,
> local, file:/tmp/1409815981504-0/testJar-1409815981505.jar) exited with code 1
> at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:837)
> at
> org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(SparkSubmitSuite.scala:311)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply$mcV$sp(SparkSubmitSuite.scala:291)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply(SparkSubmitSuite.scala:284)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$14.apply(SparkSubmitSuite.scala:284)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> ...
> - spark submit includes jars passed in through --jar *** FAILED ***
> org.apache.spark.SparkException: Process List(./bin/spark-submit, --class,
> org.apache.spark.deploy.JarCreationTest, --name, testApp, --master,
> local-cluster[2,1,512], --jars,
> file:/tmp/1409815984960-0/testJar-1409815985029.jar,file:/tmp/1409815985030-0/testJar-1409815985087.jar,
> file:/tmp/1409815984959-0/testJar-1409815984959.jar) exited with code 1
> at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:837)
> at
> org.apache.spark.deploy.SparkSubmitSuite.runSparkSubmit(SparkSubmitSuite.scala:311)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply$mcV$sp(SparkSubmitSuite.scala:305)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply(SparkSubmitSuite.scala:294)
> at
> org.apache.spark.deploy.SparkSubmitSuite$$anonfun$15.apply(SparkSubmitSuite.scala:294)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> ...
> {code}
> SBT builds don't fail, so it is likely to be due to some difference in how
> the tests are run rather than a problem with test or core project.
> This is related to http://issues.apache.org/jira/browse/SPARK-3330 but the
> cause identified in that JIRA is, at least, not the only cause. (Although, it
> wouldn't hurt to be doubly-sure this is not an issue by changing the Jenkins
> config to invoke {{mvn clean && mvn ... package}} {{mvn ... clean package}}.)
> This JIRA tracks investigation into a different cause. Right now I have some
> further information but not a PR yet.
> Part of the issue is that there is no clue in the log about why
> {{spark-submit}} exited with status 1. See
> https://github.com/apache/spark/pull/2108/files and
> https://issues.apache.org/jira/browse/SPARK-3193 for a change that would at
> least print stdout to the log too.
> The SparkSubmit program exits with 1 when the main class it is supposed to
> run is not found
> (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L322)
> This is for example SimpleApplicationTest
> (https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala#L339)
> The test actually submits an empty JAR not containing this class. It relies
> on {{spark-submit}} finding the class within the compiled test-classes of the
> Spark project. However it does seem to be compiled and present even with
> Maven.
> If modified to print stdout and s
