Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/20461#discussion_r165225508
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
---
@@ -171,7 +171,9 @@ private void failRemainingBlocks(String[]
failedBlockIds, Throwable e) {
@Override
public void onData(String streamId, ByteBuffer buf) throws IOException
{
- channel.write(buf);
+ while (buf.hasRemaining()) {
+ channel.write(buf);
--- End diff --
I don't hit this issue and it may be just because this feature is disabled
by default.
Just to be safe as
[FileDispatcherImpl](http://www.grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/sun/nio/ch/FileChannelImpl.java#210)
calls `write/pwrite` system call but doesn't check the written size. Seems the
written size may be less than the buffer size:
https://stackoverflow.com/questions/32683086/handling-incomplete-write-calls
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
