Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/3561#issuecomment-67583605
  
    Aha, I found the problem: it looks like tests in SparkSubmitSuite end up 
calling SparkSubmit.main(), which sets system properties.  I added some debug 
logging:
    
    ```diff
    --- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
    +++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
    @@ -328,6 +328,7 @@ object SparkSubmit {
         }
    
         for ((key, value) <- sysProps) {
    +      println(s"Setting system property $key $value")
           System.setProperty(key, value)
         }
    ```
    
    Now, when I run SparkSubmitSuite:
    
    ```
    [info] SparkSubmitSuite:
    [info] - prints usage on empty input (92 milliseconds)
    [info] - prints usage with only --help (4 milliseconds)
    [info] - prints error with unrecognized options (4 milliseconds)
    Setting system property SPARK_SUBMIT true
    Setting system property spark.app.name foo.jar
    Setting system property spark.jars 
file:/Users/joshrosen/Documents/spark/core/foo.jar
    Setting system property spark.master local[*]
    [info] - handle binary specified but not class (11 milliseconds)
    [info] - handles arguments with --key=val (14 milliseconds)
    [info] - handles arguments to user program (2 milliseconds)
    [info] - handles arguments to user program with name collision (2 
milliseconds)
    [info] - handles YARN cluster mode (22 milliseconds)
    [info] - handles YARN client mode (8 milliseconds)
    [info] - handles standalone cluster mode (7 milliseconds)
    [info] - handles standalone client mode (3 milliseconds)
    [info] - handles mesos client mode (3 milliseconds)
    [info] - handles confs with flag equivalents (4 milliseconds)
    ```
    
    A general fix for this issue is to use test fixtures that ensure that 
system properties which are modified in tests are restored to their old values 
after the tests finish.  I have some code to do this, so I can see about 
submitting a separate PR to add that fixture.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to