xkrogen commented on a change in pull request #34120:
URL: https://github.com/apache/spark/pull/34120#discussion_r746857642
##########
File path:
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala
##########
@@ -150,12 +152,62 @@ class YarnClusterSuite extends BaseYarnClusterSuite {
checkResult(finalState, result)
}
- test("run Spark in yarn-client mode with additional jar") {
- testWithAddJar(true)
+ test("SPARK-35672: run Spark in yarn-client mode with additional jar using
URI scheme 'local'") {
+ testWithAddJar(clientMode = true, "local")
}
- test("run Spark in yarn-cluster mode with additional jar") {
- testWithAddJar(false)
+ test("SPARK-35672: run Spark in yarn-cluster mode with additional jar using
URI scheme 'local'") {
+ testWithAddJar(clientMode = false, "local")
+ }
+
+ test("SPARK-35672: run Spark in yarn-client mode with additional jar using
URI scheme 'local' " +
+ "and gateway-replacement path") {
+ // Use the original jar URL, but set up the gateway/replacement configs
such that if
+ // replacement occurs, things will break. This ensures the replacement
doesn't apply to the
+ // driver in 'client' mode. Executors will fail in this case because they
still apply the
+ // replacement in client mode.
+ testWithAddJar(clientMode = true, "local", Some(jarUrl => {
+ (jarUrl.getPath, Map(
+ GATEWAY_ROOT_PATH.key -> Paths.get(jarUrl.toURI).getParent.toString,
+ REPLACEMENT_ROOT_PATH.key -> "/nonexistent/path/"
+ ))
+ }), expectExecutorFailure = true)
+ }
+
+ test("SPARK-35672: run Spark in yarn-cluster mode with additional jar using
URI scheme 'local' " +
Review comment:
Actually, this is exposing a legitimate bug in the older code. The
gateway replacement was done in `ExecutorRunnable`, so of course, it only
affected executors. But, in cluster mode, the driver _also_ does not run on the
gateway and thus should also have the replaced path instead of the gateway path
(which is handled in the new codepath).
But thanks for calling this out -- I forgot to include it in the PR
description. Updated now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
