GitHub user ericl opened a pull request:
https://github.com/apache/spark/pull/15306
[SPARK-17740] Spark tests should mock / interpose HDFS to ensure that
streams are closed
## What changes were proposed in this pull request?
As a followup to SPARK-17666, ensure filesystem connections are not leaked
at least in unit tests. This is done here by intercepting filesystem calls as
suggested by @JoshRosen . At the end of each test, we assert no filesystem
streams are left open.
This applies to all tests using SharedSQLContext or SharedSparkContext.
## How was this patch tested?
I verified that tests in sql and core are indeed using the filesystem
backend, and fixed the detected leaks. I also checked that reverting
https://github.com/apache/spark/pull/15245 causes many more test failures due
to connection leaks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ericl/spark sc-4672
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15306.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15306
----
commit c07b6136d2d80595612e932325b7d47ae5f36f56
Author: Eric Liang <[email protected]>
Date: 2016-09-30T01:23:57Z
Thu Sep 29 18:23:57 PDT 2016
commit a92de9626929cc4907ac789f4dd49336d0e103bd
Author: Eric Liang <[email protected]>
Date: 2016-09-30T01:33:52Z
Thu Sep 29 18:33:52 PDT 2016
commit c009783f5846669e10578492114a1a0d6631e611
Author: Eric Liang <[email protected]>
Date: 2016-09-30T01:42:03Z
Thu Sep 29 18:42:03 PDT 2016
commit 1d80d2ec396bc5225da351ff0c7e143c79000c83
Author: Eric Liang <[email protected]>
Date: 2016-09-30T01:46:00Z
Thu Sep 29 18:46:00 PDT 2016
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
