Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3739#issuecomment-68080173
Alright, I've updated this PR to remove nearly all uses of
`System.setProperty` and replace them with SparkConf.
After this, there are only a handful of cases where we set properties in
Spark core:
```
core/src/main/scala/org/apache/spark/SparkContext.scala: if (master ==
"yarn-client") System.setProperty("SPARK_YARN_MODE", "true")
core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala:
System.setProperty("spark.driver.host", "172.17.42.1") // default docker host ip
core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala:
System.setProperty("spark.driver.port", "0")
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
System.setProperty(key, value)
core/src/main/scala/org/apache/spark/deploy/history/HistoryServerArguments.scala:
System.setProperty("spark.history.fs.logDirectory", value)
core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala:
System.setProperty("spark.ui.proxyBase", proxyBase)
core/src/test/scala/org/apache/spark/SparkConfSuite.scala:
System.setProperty("spark.test.testProperty", "2")
core/src/test/scala/org/apache/spark/SparkConfSuite.scala:
System.setProperty("spark.test.testProperty", "2")
core/src/test/scala/org/apache/spark/SparkConfSuite.scala:
System.setProperty("spark.test.a", "a")
core/src/test/scala/org/apache/spark/SparkConfSuite.scala:
System.setProperty("spark.test.a.b", "a.b")
core/src/test/scala/org/apache/spark/SparkConfSuite.scala:
System.setProperty("spark.test.a.b.c", "a.b.c")
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala:
System.setProperty("spark.testing", "true")
core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala:
System.setProperty("spark.akka.frameSize", "1")
core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala:
System.setProperty("os.arch", "amd64")
core/src/test/scala/org/apache/spark/util/AkkaUtilsSuite.scala:
System.setProperty("spark.hostPort", hostname + ":" + boundPort)
core/src/test/scala/org/apache/spark/util/AkkaUtilsSuite.scala:
System.setProperty("spark.hostPort", hostname + ":" + boundPort)
core/src/test/scala/org/apache/spark/util/AkkaUtilsSuite.scala:
System.setProperty("spark.hostPort", hostname + ":" + boundPort)
core/src/test/scala/org/apache/spark/util/AkkaUtilsSuite.scala:
System.setProperty("spark.hostPort", hostname + ":" + boundPort)
core/src/test/scala/org/apache/spark/util/SizeEstimatorSuite.scala:
System.setProperty("os.arch", "amd64")
core/src/test/scala/org/apache/spark/util/SizeEstimatorSuite.scala:
System.setProperty("spark.test.useCompressedOops", "true")
core/src/test/scala/org/apache/spark/util/SizeEstimatorSuite.scala:
System.setProperty("os.arch", "x86")
core/src/test/scala/org/apache/spark/util/SizeEstimatorSuite.scala:
System.setProperty("os.arch", "amd64")
core/src/test/scala/org/apache/spark/util/SizeEstimatorSuite.scala:
System.setProperty("spark.test.useCompressedOops", "false")
core/src/test/scala/org/apache/spark/util/UtilsSuite.scala:
System.setProperty("spark.test.fileNameLoadB", "2")
```
Going through these:
- FaultToleranceTest isn't a regular unit test.
- SparkSubmit sets some properties, but I've guarded against this with the
ResetSystemProperties mixin.
- SparkConfSuite needs to set properties since it's testing SparkConf
itself.
- Yarn and the HistoryServer, similarly, need to set properties.
- AkkaUtilsSuite is relatively new and actually uses SparkConf, so I assume
there's a valid reason for it's use of System.setProperty.
- BlockManagerSuite and SizeEstimatorSuite may need to set the architecture
property because third-party code reads it.
I'd love to get some additional eyes on these changes just to make sure
that I haven't inadvertently broken anything. I intend to merge this into all
of the backport branches.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
