Github user advancedxy commented on a diff in the pull request:
https://github.com/apache/spark/pull/21327#discussion_r188532109
--- Diff:
core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala ---
@@ -63,15 +63,18 @@ private[spark] class ChunkedByteBuffer(var chunks:
Array[ByteBuffer]) {
*/
def writeFully(channel: WritableByteChannel): Unit = {
for (bytes <- getChunks()) {
- val curChunkLimit = bytes.limit()
+ val originalLimit = bytes.limit()
while (bytes.hasRemaining) {
- try {
- val ioSize = Math.min(bytes.remaining(), bufferWriteChunkSize)
- bytes.limit(bytes.position() + ioSize)
- channel.write(bytes)
- } finally {
- bytes.limit(curChunkLimit)
- }
+ // If `bytes` is an on-heap ByteBuffer, the JDK will copy it to a
temporary direct
+ // ByteBuffer when writing it out. The JDK caches one temporary
buffer per thread, and we
--- End diff --
> The JDK caches one temporary buffer per thread
I don't think this statement is correct. According to
[Util.java](http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/sun/nio/ch/Util.java#l48),
the cached number of temporary direct buffer is up to `IOUtil.IOV_MAX`.
> if the cached temp buffer gets created and freed frequently
The problem is that the varied-sized heap buffers could cause a new
allocation of temporary direct buffer and free of old direct buffer if the
buffer size is larger than before
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
