GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/2974
[SPARK-4107] Fix incorrect handling of read() and skip() return values
(branch-1.1 backport)
`read()` may return fewer bytes than requested; when this occurred, the old
code would silently return less data than requested, which might cause stream
corruption errors. `skip()` faces similar issues, too.
This patch fixes several cases where we mis-handle these methods' return
values.
This is a backport of #2969 to `branch-1.1`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark spark-4107-branch-1.1-backport
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2974.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2974
----
commit d82c05b05f38828e95a67ab1ac6691077bbda465
Author: Josh Rosen <[email protected]>
Date: 2014-10-28T07:04:16Z
[SPARK-4107] Fix incorrect handling of read() and skip() return values
`read()` may return fewer bytes than requested; when this occurred, the old
code would silently return less data than requested, which might cause stream
corruption errors. `skip()` faces similar issues, too.
This patch fixes several cases where we mis-handle these methods' return
values.
Author: Josh Rosen <[email protected]>
Closes #2969 from JoshRosen/file-channel-read-fix and squashes the
following commits:
e724a9f [Josh Rosen] Fix similar issue of not checking skip() return value.
cbc03ce [Josh Rosen] Update the other log message, too.
01e6015 [Josh Rosen] file.getName -> file.getAbsolutePath
d961d95 [Josh Rosen] Fix another issue in FileServerSuite.
b9265d2 [Josh Rosen] Fix a similar (minor) issue in TestUtils.
cd9d76f [Josh Rosen] Fix a similar error in Tachyon:
3db0008 [Josh Rosen] Fix a similar read() error in Utils.offsetBytes().
db985ed [Josh Rosen] Fix unsafe usage of FileChannel.read():
Conflicts:
core/src/main/scala/org/apache/spark/network/ManagedBuffer.scala
core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockManager.scala
core/src/main/scala/org/apache/spark/storage/DiskStore.scala
core/src/test/scala/org/apache/spark/FileServerSuite.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
