zhoukang created PARQUET-1320:
---------------------------------
Summary: Fast clean unused direct memory when decompress
Key: PARQUET-1320
URL: https://issues.apache.org/jira/browse/PARQUET-1320
Project: Parquet
Issue Type: Improvement
Components: parquet-mr
Reporter: zhoukang
When use *NonBlockedDecompressorStream* which call:
*SnappyDecompressor.setInput*
{code:java}
public synchronized void setInput(byte[] buffer, int off, int len) {
SnappyUtil
public synchronized void setInput(byte[] buffer, int off, int len) {
SnappyUtil.validateBuffer(buffer, off, len);
if (inputBuffer.capacity() - inputBuffer.position() < len) {
ByteBuffer newBuffer = ByteBuffer.allocateDirect(inputBuffer.position() + len);
inputBuffer.rewind();
newBuffer.put(inputBuffer);
inputBuffer = newBuffer;
} else {
inputBuffer.limit(inputBuffer.position() + len);
}
inputBuffer.put(buffer, off, len);
}
.validateBuffer(buffer, off, len);
if (inputBuffer.capacity() - inputBuffer.position() < len) {
ByteBuffer newBuffer = ByteBuffer.allocateDirect(inputBuffer.position() + len);
inputBuffer.rewind();
newBuffer.put(inputBuffer);
inputBuffer = newBuffer;
} else {
inputBuffer.limit(inputBuffer.position() + len);
}
inputBuffer.put(buffer, off, len);
}
{code}
If we do not get any full gc for old gen.we may failed by off-heap memory leak
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
