GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/20144
[SPARK-21475][CORE][2nd attempt] Change to use NIO's Files API for external
shuffle service
## What changes were proposed in this pull request?
This PR is the second attempt of #18684 , NIO's Files API doesn't override
`skip` method for `InputStream`, so it will bring in performance issue
(mentioned in #20119). But using `FileInputStream`/`FileOutputStream` will also
bring in memory issue
(https://dzone.com/articles/fileinputstream-fileoutputstream-considered-harmful),
which is severe for long running external shuffle service. So here in this
proposal, only fixing the external shuffle service related code.
## How was this patch tested?
Existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-21475-v2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20144.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20144
----
commit 277b801b3ccee95d8f30c02b1ba8693ab39f302a
Author: jerryshao <sshao@...>
Date: 2018-01-04T01:48:55Z
Change to NIO's file API for external shuffle service
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
