SabrinaZhaozyf opened a new issue, #13580:
URL: https://github.com/apache/pinot/issues/13580
We have observed an issue with direct buffer deallocation in `DataFetcher`.
Specifically, `ForwardIndexReaderContext` (`ChunkReaderContext` in the
following case) is not being closed explicitly in `DataFetcher` , meaning that
we are solely replying on gc to clear the direct buffers.
Impact: This has caused OOM on our query path when reading forward index on
a large string/JSON column.
```
ERROR Caught serious error while processing query: ..
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[?:?]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
at
org.apache.pinot.segment.local.segment.index.readers.forward.ChunkReaderContext.<init>(ChunkReaderContext.java:49)
at
org.apache.pinot.segment.local.segment.index.readers.forward.VarByteChunkSVForwardIndexReader.createContext(VarByteChunkSVForwardIndexReader.java:55)
at
org.apache.pinot.segment.local.segment.index.readers.forward.VarByteChunkSVForwardIndexReader.createContext(VarByteChunkSVForwardIndexReader.java:38)
at
org.apache.pinot.core.common.DataFetcher$ColumnValueReader.getReaderContext(DataFetcher.java:464)
at
org.apache.pinot.core.common.DataFetcher$ColumnValueReader.readStringValues(DataFetcher.java:567)
at
org.apache.pinot.core.common.DataFetcher.fetchStringValues(DataFetcher.java:239)
at
org.apache.pinot.core.common.DataBlockCache.getStringValuesForSVColumn(DataBlockCache.java:277)
at
org.apache.pinot.core.operator.docvalsets.ProjectionBlockValSet.getStringValuesSV(ProjectionBlockValSet.java:153)
at
org.apache.pinot.core.common.RowBasedBlockValueFetcher.createFetcher(RowBasedBlockValueFetcher.java:67)
at
org.apache.pinot.core.common.RowBasedBlockValueFetcher.<init>(RowBasedBlockValueFetcher.java:33)
at
org.apache.pinot.core.operator.query.SelectionOrderByOperator.computePartiallyOrdered(SelectionOrderByOperator.java:280)
at
org.apache.pinot.core.operator.query.SelectionOrderByOperator.getNextBlock(SelectionOrderByOperator.java:132)
at
org.apache.pinot.core.operator.query.SelectionOrderByOperator.getNextBlock(SelectionOrderByOperator.java:73)
at
org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:43)
at
org.apache.pinot.core.operator.combine.MinMaxValueBasedSelectionOrderByCombineOperator.processSegments(MinMaxValueBasedSelectionOrderByCombineOperator.java:207)
```
We are able to reproduce the issue and verified that direct memory, once
allocated, will not be released until GC kicks in and usage can consistently
stay high even 1-2 hrs after we stopped querying (see the following gc and
memory usage correlation).
This resource leak has also been called out as a TODO in
https://github.com/apache/pinot/pull/5625.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
