Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3561#issuecomment-67583605
Aha, I found the problem: it looks like tests in SparkSubmitSuite end up
calling SparkSubmit.main(), which sets system properties. I added some debug
logging:
```diff
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -328,6 +328,7 @@ object SparkSubmit {
}
for ((key, value) <- sysProps) {
+ println(s"Setting system property $key $value")
System.setProperty(key, value)
}
```
Now, when I run SparkSubmitSuite:
```
[info] SparkSubmitSuite:
[info] - prints usage on empty input (92 milliseconds)
[info] - prints usage with only --help (4 milliseconds)
[info] - prints error with unrecognized options (4 milliseconds)
Setting system property SPARK_SUBMIT true
Setting system property spark.app.name foo.jar
Setting system property spark.jars
file:/Users/joshrosen/Documents/spark/core/foo.jar
Setting system property spark.master local[*]
[info] - handle binary specified but not class (11 milliseconds)
[info] - handles arguments with --key=val (14 milliseconds)
[info] - handles arguments to user program (2 milliseconds)
[info] - handles arguments to user program with name collision (2
milliseconds)
[info] - handles YARN cluster mode (22 milliseconds)
[info] - handles YARN client mode (8 milliseconds)
[info] - handles standalone cluster mode (7 milliseconds)
[info] - handles standalone client mode (3 milliseconds)
[info] - handles mesos client mode (3 milliseconds)
[info] - handles confs with flag equivalents (4 milliseconds)
```
A general fix for this issue is to use test fixtures that ensure that
system properties which are modified in tests are restored to their old values
after the tests finish. I have some code to do this, so I can see about
submitting a separate PR to add that fixture.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]