LuciferYang edited a comment on pull request #32648: URL: https://github.com/apache/spark/pull/32648#issuecomment-961174098
@srowen @dongjoon-hyun Yes, master branch using Scala 2.12 and Scala 2.13 will have the same problem. Our Jenkins maven job always executes `mvn clean package -DskipTests` before `mvn test`, so it won't reproduce this problem. I think the key point here is `assembly/target/scala-%s/jars`. `mvn clean package -DskipTests` command will produce this dir, Its content is almost the same as `spark-client/jars`, `buildClassPath` method in `o.a.s.launcher.AbstractCommandBuilder` will add this dir to classpath, this behavior will avoid the above exceptions. To prove this I did the following tests: 1. `build/mvn clean package -DskipTests -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive` to produce `assembly/target/scala-2.12/jars` 2. Delete `assembly/target/scala-2.12/jars` manually 3. `build/mvn test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 -pl resource-managers/yarn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnShuffleIntegrationSuite` `external shuffle service` in `YarnShuffleIntegrationSuite` will fail for the same reason. Then why does this case rely on `Spark ML`? I haven't investigated it clearly. In short, this problem has existed before this pr. I think we can ignore it from this PR and I created a new jira(https://issues.apache.org/jira/browse/SPARK-37209) to continue tracking it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
