Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/21593#discussion_r196947295 --- Diff: common/network-common/src/main/java/org/apache/spark/network/protocol/MessageWithHeader.java --- @@ -137,30 +137,15 @@ protected void deallocate() { } private int copyByteBuf(ByteBuf buf, WritableByteChannel target) throws IOException { - ByteBuffer buffer = buf.nioBuffer(); - int written = (buffer.remaining() <= NIO_BUFFER_LIMIT) ? - target.write(buffer) : writeNioBuffer(target, buffer); + // SPARK-24578: cap the sub-region's size of returned nio buffer to improve the performance + // for the case that the passed-in buffer has too many components. + int length = Math.min(buf.readableBytes(), NIO_BUFFER_LIMIT); + ByteBuffer buffer = buf.nioBuffer(buf.readerIndex(), length); --- End diff -- Fair enough. I just spent several minutes to write the following codes: ``` private int copyByteBuf(ByteBuf buf, WritableByteChannel target) throws IOException { // SPARK-24578: cap the sub-region's size of returned nio buffer to improve the performance // for the case that the passed-in buffer has too many components. int length = Math.min(buf.readableBytes(), NIO_BUFFER_LIMIT); ByteBuffer[] buffers = buf.nioBuffers(buf.readerIndex(), length); int totalWritten = 0; for (ByteBuffer buffer : buffers) { int remaining = buffer.remaining(); int written = target.write(buffer); totalWritten += written; if (written < remaining) { break; } } buf.skipBytes(totalWritten); return totalWritten; } ``` Feel free to use them in your follow up PR.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org