Xiao Chen created HDFS-13540: -------------------------------- Summary: DFSStripedInputStream should not allocate new buffers during close / unbuffer Key: HDFS-13540 URL: https://issues.apache.org/jira/browse/HDFS-13540 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Xiao Chen Assignee: Xiao Chen
This was found in the same scenario where HDFS-13539 is caught. There are 2 OOM that looks interesting: {noformat} FSDataInputStream#close error: OutOfMemoryError: Direct buffer memoryjava.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:694) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) at org.apache.hadoop.io.ElasticByteBufferPool.getBuffer(ElasticByteBufferPool.java:95) at org.apache.hadoop.hdfs.DFSStripedInputStream.resetCurStripeBuffer(DFSStripedInputStream.java:118) at org.apache.hadoop.hdfs.DFSStripedInputStream.closeCurrentBlockReaders(DFSStripedInputStream.java:205) at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:672) at org.apache.hadoop.hdfs.DFSStripedInputStream.close(DFSStripedInputStream.java:181) at java.io.FilterInputStream.close(FilterInputStream.java:181) {noformat} and {noformat} org/apache/hadoop/fs/FSDataInputStream#unbuffer failed: error: OutOfMemoryError: Direct buffer memoryjava.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:694) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) at org.apache.hadoop.io.ElasticByteBufferPool.getBuffer(ElasticByteBufferPool.java:95) at org.apache.hadoop.hdfs.DFSStripedInputStream.resetCurStripeBuffer(DFSStripedInputStream.java:118) at org.apache.hadoop.hdfs.DFSStripedInputStream.closeCurrentBlockReaders(DFSStripedInputStream.java:205) at org.apache.hadoop.hdfs.DFSInputStream.unbuffer(DFSInputStream.java:1782) at org.apache.hadoop.fs.StreamCapabilitiesPolicy.unbuffer(StreamCapabilitiesPolicy.java:48) at org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:230) {noformat} As the stack trace goes, {{resetCurStripeBuffer}} will get buffer from the buffer pool. We could save the cost of doing so if it's just a close or unbuffer call. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org