Re: [I] [Bug] 0.9.0 When querying the paimon table in Parquet format through starrocks, the following error appears. I tried to adjust the heap memory size, but it still works like this [paimon]

Thu, 17 Apr 2025 04:55:07 -0700 


guoxiaolongzte commented on issue #5469:
URL: https://github.com/apache/paimon/issues/5469#issuecomment-2812633806

   I tried https://github.com/apache/paimon/pull/5421, but there are still 
problems.
   [13.443s][warning][gc,alloc] Thread-24: Retried waiting for GCLocker too 
often allocating 1048578 words
   Exception in thread "Thread-24" java.lang.OutOfMemoryError: Java heap space
           at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:64)
           at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:363)
           at 
org.apache.paimon.shade.org.apache.parquet.bytes.HeapByteBufferAllocator.allocate(HeapByteBufferAllocator.java:34)
           at 
org.apache.paimon.shade.org.apache.parquet.hadoop.ParquetFileReader$ConsecutivePartList.readAll(ParquetFileReader.java:1550)
           at 
org.apache.paimon.shade.org.apache.parquet.hadoop.ParquetFileReader.readAllPartsVectoredOrNormal(ParquetFileReader.java:578)
           at 
org.apache.paimon.shade.org.apache.parquet.hadoop.ParquetFileReader.internalReadRowGroup(ParquetFileReader.java:471)
           at 
org.apache.paimon.shade.org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:420)
           at 
org.apache.paimon.format.parquet.ParquetReaderFactory$ParquetReader.readNextRowGroup(ParquetReaderFactory.java:348)
           at 
org.apache.paimon.format.parquet.ParquetReaderFactory$ParquetReader.nextBatch(ParquetReaderFactory.java:327)
           at 
org.apache.paimon.format.parquet.ParquetReaderFactory$ParquetReader.readBatch(ParquetReaderFactory.java:309)
           at 
org.apache.paimon.io.FileRecordReader.readBatch(FileRecordReader.java:47)
           at 
org.apache.paimon.io.KeyValueDataFileRecordReader.readBatch(KeyValueDataFileRecordReader.java:48)
           at 
org.apache.paimon.mergetree.compact.ConcatRecordReader.readBatch(ConcatRecordReader.java:66)
           at 
org.apache.paimon.mergetree.compact.LoserTree$LeafIterator.advanceIfAvailable(LoserTree.java:315)
           at 
org.apache.paimon.mergetree.compact.LoserTree.initializeIfNeeded(LoserTree.java:87)
           at 
org.apache.paimon.mergetree.compact.SortMergeReaderWithLoserTree.readBatch(SortMergeReaderWithLoserTree.java:71)
           at 
org.apache.paimon.mergetree.DropDeleteReader.readBatch(DropDeleteReader.java:44)
           at 
org.apache.paimon.reader.RecordReader$1.readBatch(RecordReader.java:173)
           at 
org.apache.paimon.table.source.KeyValueTableRead$1.readBatch(KeyValueTableRead.java:131)
           at 
org.apache.paimon.reader.RecordReader$2.readBatch(RecordReader.java:194)
           at 
org.apache.paimon.reader.RecordReaderIterator.<init>(RecordReaderIterator.java:37)
           at 
com.starrocks.paimon.reader.PaimonSplitScanner.initReader(PaimonSplitScanner.java:106)
           at 
com.starrocks.paimon.reader.PaimonSplitScanner.open(PaimonSplitScanner.java:115)
   Exception in thread "Thread-47" java.lang.OutOfMemoryError: Java heap space
           at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:64)
           at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:363)
           at 
org.apache.paimon.shade.org.apache.parquet.bytes.HeapByteBufferAllocator.allocate(HeapByteBufferAllocator.java:34)
           at 
org.apache.paimon.shade.org.apache.parquet.hadoop.ParquetFileReader$ConsecutivePartList.readAll(ParquetFileReader.java:1550)
           at 
org.apache.paimon.shade.org.apache.parquet.hadoop.ParquetFileReader.readAllPartsVectoredOrNormal(ParquetFileReader.java:578)
           at 
org.apache.paimon.shade.org.apache.parquet.hadoop.ParquetFileReader.internalReadRowGroup(ParquetFileReader.java:471)
           at 
org.apache.paimon.shade.org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:420)
           at 
org.apache.paimon.format.parquet.ParquetReaderFactory$ParquetReader.readNextRowGroup(ParquetReaderFactory.java:348)
           at 
org.apache.paimon.format.parquet.ParquetReaderFactory$ParquetReader.nextBatch(ParquetReaderFactory.java:327)
           at 
org.apache.paimon.format.parquet.ParquetReaderFactory$ParquetReader.readBatch(ParquetReaderFactory.java:309)
           at 
org.apache.paimon.io.FileRecordReader.readBatch(FileRecordReader.java:47)
           at 
org.apache.paimon.io.KeyValueDataFileRecordReader.readBatch(KeyValueDataFileRecordReader.java:48)
           at 
org.apache.paimon.mergetree.compact.ConcatRecordReader.readBatch(ConcatRecordReader.java:66)
           at 
org.apache.paimon.mergetree.compact.LoserTree$LeafIterator.advanceIfAvailable(LoserTree.java:315)
           at 
org.apache.paimon.mergetree.compact.LoserTree.initializeIfNeeded(LoserTree.java:87)
           at 
org.apache.paimon.mergetree.compact.SortMergeReaderWithLoserTree.readBatch(SortMergeReaderWithLoserTree.java:71)
           at 
org.apache.paimon.mergetree.DropDeleteReader.readBatch(DropDeleteReader.java:44)
           at 
org.apache.paimon.reader.RecordReader$1.readBatch(RecordReader.java:173)
           at 
org.apache.paimon.table.source.KeyValueTableRead$1.readBatch(KeyValueTableRead.java:131)
           at 
org.apache.paimon.reader.RecordReader$2.readBatch(RecordReader.java:194)
           at 
org.apache.paimon.reader.RecordReaderIterator.<init>(RecordReaderIterator.java:37)
           at 
com.starrocks.paimon.reader.PaimonSplitScanner.initReader(PaimonSplitScanner.java:106)
           at 
com.starrocks.paimon.reader.PaimonSplitScanner.open(PaimonSplitScanner.java:115)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@paimon.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Reply via email to