LuciferYang edited a comment on pull request #32648:
URL: https://github.com/apache/spark/pull/32648#issuecomment-961174098
@srowen @dongjoon-hyun Yes, master branch using Scala 2.12 and Scala 2.13
will have the same problem.
Our Jenkins maven job always executes `mvn clean package -DskipTests` before
`mvn test`, so it won't reproduce this problem.
I think the key point here is `assembly/target/scala-%s/jars`.
`mvn clean package -DskipTests` command will produce this dir, Its content
is almost the same as `spark-client/jars`, `buildClassPath` method in
`o.a.s.launcher.AbstractCommandBuilder` will add this dir to classpath, this
behavior will avoid the above exceptions.
To prove this I did the following tests:
1. `build/mvn clean package -DskipTests -Phadoop-3.2 -Phive-2.3
-Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver
-Pspark-ganglia-lgpl -Pkubernetes -Phive` to produce
`assembly/target/scala-2.12/jars`
2. Delete `assembly/target/scala-2.12/jars` manually
3. `build/mvn test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn
-Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
-Pscala-2.13 -pl resource-managers/yarn -Dtest=none
-DwildcardSuites=org.apache.spark.deploy.yarn.YarnShuffleIntegrationSuite`
`external shuffle service` in `YarnShuffleIntegrationSuite` will fail for
the same reason.
Then why does this case rely on `Spark ML`? I haven't investigated it
clearly. In short, this problem has existed before this pr. I think we can
ignore it from this PR and I created a new jira(SPARK-37209) to continue
tracking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
