GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/3739
[SPARK-4893] Add test fixture for resetting system properties after
individual test runs
Several of our tests call System.setProperty (or test code which implicitly
sets system properties) and don't always reset/clear the modified properties,
which can create ordering dependencies between tests and cause hard-to-diagnose
failures.
To address this, this PR introduces a `ResetSystemProperties` ScalaTest
mixin class which uses `withFixture` to snapshot the system properties before
individual tests and to automatically restore them on test completion / failure.
I've updated the Spark core test suites to use this and found several
instances where we didn't properly clean up properties. I'd like to apply
similar fixes to the other subprojects, too; I can't directly use the
`ResetSystemProperties` class from Spark Core because it needs to be in a test
JAR in order to compile against ScalaTest, and we don't have the build
configured so that subprojects can depend on core's tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark
cleanup-system-properties-in-tests
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3739.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3739
----
commit 4dcea388c9baa92c9564ef7a97792ddb938bc5b1
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:14:01Z
Move withSystemProperty to TestUtils class.
commit 9e3e0ddff745f6aa2b248d5cbcc6cd7945339368
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:14:25Z
Add ResetSystemProperties test fixture mixin; use it in SparkSubmitSuite.
commit 628f46c4839cd9e4a2563ba772f1cfeea1e2d316
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:18:28Z
Use ResetSystemProperties in DistributedSuite
commit 14a92e4b5f624ab23e52bf68153445f14029d012
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:23:38Z
Use withSystemProperty in FileServerSuite
commit 60a63a1ce24949665374b53c6cd3d557df99f49c
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:25:11Z
Use ResetSystemProperties in JobCancellationSuite
commit 51aa87038690daecbe8db6fb575b4e597043c649
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:26:14Z
Use withSystemProperty in ShuffleSuite
commit c83ded805d287eb41c4eeb2c4d24badb258412a1
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:28:38Z
Use ResetSystemProperties in SparkConfSuite
commit 0995c4b0ff5a650116824cffe247378f987a8375
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:30:13Z
Use ResetSystemProperties in SparkContextSchedulerCreationSuite
commit 5b3cb548f3ec0fb31718285d6b8473c9fcdc219d
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:32:12Z
Use ResetSystemProperties in SparkListenerSuite
commit e9ded626a3b10fbe354c98e62fae275180994f8b
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:33:25Z
Use ResetSystemProperties in TaskSchedulerImplSuite
commit b0daff2eb3e9ba6ed914e22e57f9448ad2fc6334
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:35:27Z
Use ResetSystemProperties in BlockManagerSuite
commit dd9492bb5f8c51e7b4ae8c422c00655c3632f510
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:36:08Z
Use ResetSystemProperties in AkkaUtilsSuite
commit 1d1aa5a19bd8ce1f49b790dda961e0616484d996
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:38:43Z
Use ResetSystemProperties in SizeEstimatorSuite
commit 25bfce2a31e835479400d31718d384156d27462d
Author: Josh Rosen <[email protected]>
Date: 2014-12-19T01:39:42Z
Use ResetSystemProperties in UtilsSuite
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
