zhoukang created SPARK-21527:
--------------------------------
Summary: Use buffer limit in order to use JAVA NIO Util's
buffercache
Key: SPARK-21527
URL: https://issues.apache.org/jira/browse/SPARK-21527
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.2.0
Reporter: zhoukang
Right now, ChunkedByteBuffer#writeFully do not slice bytes first.We observe
code in java nio Util below:
{code:java}
public static ByteBuffer More ...getTemporaryDirectBuffer(int size) {
BufferCache cache = bufferCache.get();
ByteBuffer buf = cache.get(size);
if (buf != null) {
return buf;
} else {
// No suitable buffer in the cache so we need to allocate a new
// one. To avoid the cache growing then we remove the first
// buffer from the cache and free it.
if (!cache.isEmpty()) {
buf = cache.removeFirst();
free(buf);
}
return ByteBuffer.allocateDirect(size);
}
}
{code}
If we slice first with a fixed size, we can use buffer cache and only need to
allocate at the first write call.
Since we allocate new buffer, we can not control the free time of this
buffer.This once cause memory issue in our production cluster.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
