Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/16989#discussion_r165193691
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
---
@@ -126,4 +150,38 @@ private void failRemainingBlocks(String[]
failedBlockIds, Throwable e) {
}
}
}
+
+ private class DownloadCallback implements StreamCallback {
+
+ private WritableByteChannel channel = null;
+ private File targetFile = null;
+ private int chunkIndex;
+
+ public DownloadCallback(File targetFile, int chunkIndex) throws
IOException {
+ this.targetFile = targetFile;
+ this.channel = Channels.newChannel(new FileOutputStream(targetFile));
+ this.chunkIndex = chunkIndex;
+ }
+
+ @Override
+ public void onData(String streamId, ByteBuffer buf) throws IOException
{
+ channel.write(buf);
--- End diff --
@squito This is a Java Channel. Not sure how to call
`io.netty.channel.epoll.AbstractEpollStreamChannel.spliceTo` here.
By the way, I think this is a zero-copy transfer since the underlying
buffer is an off heap buffer.
Anyway, I found a bug here...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
