xkrogen commented on a change in pull request #34120:
URL: https://github.com/apache/spark/pull/34120#discussion_r746998575
##########
File path:
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala
##########
@@ -150,12 +152,62 @@ class YarnClusterSuite extends BaseYarnClusterSuite {
checkResult(finalState, result)
}
- test("run Spark in yarn-client mode with additional jar") {
- testWithAddJar(true)
+ test("SPARK-35672: run Spark in yarn-client mode with additional jar using
URI scheme 'local'") {
+ testWithAddJar(clientMode = true, "local")
}
- test("run Spark in yarn-cluster mode with additional jar") {
- testWithAddJar(false)
+ test("SPARK-35672: run Spark in yarn-cluster mode with additional jar using
URI scheme 'local'") {
+ testWithAddJar(clientMode = false, "local")
+ }
+
+ test("SPARK-35672: run Spark in yarn-client mode with additional jar using
URI scheme 'local' " +
+ "and gateway-replacement path") {
+ // Use the original jar URL, but set up the gateway/replacement configs
such that if
+ // replacement occurs, things will break. This ensures the replacement
doesn't apply to the
+ // driver in 'client' mode. Executors will fail in this case because they
still apply the
+ // replacement in client mode.
+ testWithAddJar(clientMode = true, "local", Some(jarUrl => {
+ (jarUrl.getPath, Map(
+ GATEWAY_ROOT_PATH.key -> Paths.get(jarUrl.toURI).getParent.toString,
+ REPLACEMENT_ROOT_PATH.key -> "/nonexistent/path/"
+ ))
+ }), expectExecutorFailure = true)
+ }
+
+ test("SPARK-35672: run Spark in yarn-cluster mode with additional jar using
URI scheme 'local' " +
+ "and gateway-replacement path") {
+ // Put a prefix in front of the original jar URL which causes it to be an
invalid path.
+ // Set up the gateway/replacement configs such that if replacement occurs,
it is a valid
+ // path again (by removing the prefix). This ensures the replacement is
applied.
+ val gatewayPath = "/replaceme/nonexistent/"
+ testWithAddJar(clientMode = false, "local", Some(jarUrl => {
+ (gatewayPath + jarUrl.getPath, Map(
+ GATEWAY_ROOT_PATH.key -> gatewayPath,
+ REPLACEMENT_ROOT_PATH.key -> ""
+ ))
+ }))
+ }
+
+ test("SPARK-35672: run Spark in yarn-client mode with additional jar using
URI scheme 'local' " +
Review comment:
Yes, I originally used this approach because I was having trouble
passing environment variables to the spawned job. I finally realized that it
was because I was trying to use the `extraEnv` argument on `runSpark`, which
sets environment variables for `spark-submit`, not the job itself. I've updated
this to (a) fail if gateway string replacement doesn't happen, and (b) actually
replace environment variables with a new string, instead of an empty string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
