GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/18091
[SPARK-20868][CORE] UnsafeShuffleWriter should verify the position after
FileChannel.transferTo
## What changes were proposed in this pull request?
Long time ago we fixed a
[bug](https://issues.apache.org/jira/browse/SPARK-3948) in shuffle writer about
`FileChannel.transferTo`. We were not very confident about that fix, so we
added a position check after the writing, try to discover the bug earlier.
However this checking is missing in the new `UnsafeShuffleWriter`, this PR
adds it.
https://issues.apache.org/jira/browse/SPARK-18105 maybe related to that
`FileChannel.transferTo` bug, hopefully we can find out the root cause after
adding this position check.
## How was this patch tested?
N/A
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark shuffle
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18091.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18091
----
commit fb3b31e7da2c584d2b319696cc5dfcef378fc470
Author: Wenchen Fan <[email protected]>
Date: 2017-05-24T16:21:29Z
UnsafeShuffleWriter should verify the position after FileChannel.transferTo
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
