tsreaper opened a new pull request, #9183:
URL: https://github.com/apache/paimon/pull/9183
### Purpose
Currently when reading spilled data, there is a small probability to meet
the following exception.
```
Caused by: org.apache.paimon.compression.BufferDecompressionException: Input
is corrupted, invalid length.
at
org.apache.paimon.compression.CompressorUtils.validateLength(CompressorUtils.java:49)
~[?:?]
at
org.apache.paimon.compression.Lz4BlockDecompressor.decompress(Lz4BlockDecompressor.java:48)
~[?:?]
at
org.apache.paimon.disk.ChannelReaderInputView.nextSegment(ChannelReaderInputView.java:83)
~[?:?]
at
org.apache.paimon.data.AbstractPagedInputView.doAdvance(AbstractPagedInputView.java:153)
~[?:?]
at
org.apache.paimon.data.AbstractPagedInputView.advance(AbstractPagedInputView.java:147)
~[?:?]
at
org.apache.paimon.data.AbstractPagedInputView.readInt(AbstractPagedInputView.java:322)
~[?:?]
at
org.apache.paimon.data.serializer.BinaryRowSerializer.deserialize(BinaryRowSerializer.java:77)
~[?:?]
at
org.apache.paimon.disk.ChannelReaderInputViewIterator.next(ChannelReaderInputViewIterator.java:63)
~[?:?]
at
org.apache.paimon.mergetree.MergeSorter$ChannelReaderReader$1.next(MergeSorter.java:257)
~[?:?]
at
org.apache.paimon.mergetree.MergeSorter$ChannelReaderReader$1.next(MergeSorter.java:254)
~[?:?]
at
org.apache.paimon.mergetree.compact.LoserTree$LeafIterator.advanceIfAvailable(LoserTree.java:320)
~[?:?]
at
org.apache.paimon.mergetree.compact.LoserTree.initializeIfNeeded(LoserTree.java:87)
~[?:?]
at
org.apache.paimon.mergetree.compact.SortMergeReaderWithLoserTree.readBatch(SortMergeReaderWithLoserTree.java:81)
~[?:?]
at
org.apache.paimon.reader.RecordReaderIterator.<init>(RecordReaderIterator.java:37)
~[?:?]
at
org.apache.paimon.reader.RecordReader.toCloseableIterator(RecordReader.java:210)
~[?:?]
at
org.apache.paimon.mergetree.compact.ChangelogMergeTreeRewriter.rewriteOrProduceChangelog(ChangelogMergeTreeRewriter.java:133)
~[?:?]
at
org.apache.paimon.mergetree.compact.ChangelogMergeTreeRewriter.rewrite(ChangelogMergeTreeRewriter.java:106)
~[?:?]
at
org.apache.paimon.mergetree.compact.MergeTreeCompactTask.rewriteImpl(MergeTreeCompactTask.java:162)
~[?:?]
at
org.apache.paimon.mergetree.compact.MergeTreeCompactTask.rewrite(MergeTreeCompactTask.java:157)
~[?:?]
at
org.apache.paimon.mergetree.compact.MergeTreeCompactTask.doCompact(MergeTreeCompactTask.java:111)
~[?:?]
at org.apache.paimon.compact.CompactTask.call(CompactTask.java:55) ~[?:?]
at org.apache.paimon.compact.CompactTask.call(CompactTask.java:34) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
... 1 more
```
There are two possible causes:
1. `BufferFileChannelReader` does not fully read the byte buffer.
2. Empty spilled block is not recorded correctly. It is possible to produce
an empty spilled block if logically there is no data (for example when we
delete all records in a bucket without full compaction).
This PR fixes both causes.
### Tests
* `ChannelWriterOutputViewTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
