Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21593#discussion_r196947295
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/protocol/MessageWithHeader.java
---
@@ -137,30 +137,15 @@ protected void deallocate() {
}
private int copyByteBuf(ByteBuf buf, WritableByteChannel target) throws
IOException {
- ByteBuffer buffer = buf.nioBuffer();
- int written = (buffer.remaining() <= NIO_BUFFER_LIMIT) ?
- target.write(buffer) : writeNioBuffer(target, buffer);
+ // SPARK-24578: cap the sub-region's size of returned nio buffer to
improve the performance
+ // for the case that the passed-in buffer has too many components.
+ int length = Math.min(buf.readableBytes(), NIO_BUFFER_LIMIT);
+ ByteBuffer buffer = buf.nioBuffer(buf.readerIndex(), length);
--- End diff --
Fair enough.
I just spent several minutes to write the following codes:
```
private int copyByteBuf(ByteBuf buf, WritableByteChannel target) throws
IOException {
// SPARK-24578: cap the sub-region's size of returned nio buffer to
improve the performance
// for the case that the passed-in buffer has too many components.
int length = Math.min(buf.readableBytes(), NIO_BUFFER_LIMIT);
ByteBuffer[] buffers = buf.nioBuffers(buf.readerIndex(), length);
int totalWritten = 0;
for (ByteBuffer buffer : buffers) {
int remaining = buffer.remaining();
int written = target.write(buffer);
totalWritten += written;
if (written < remaining) {
break;
}
}
buf.skipBytes(totalWritten);
return totalWritten;
}
```
Feel free to use them in your follow up PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
