richardstartin opened a new pull request #8286:
URL: https://github.com/apache/pinot/pull/8286
This addresses a bug report of a buffer overflow creating MV string raw
forward index, with the following stack trace:
```
Start building IndexCreator!
Could not build segment
java.lang.IllegalArgumentException: newPosition > limit: (1048580 > 1048575)
at java.nio.Buffer.createPositionException(Buffer.java:318) ~[?:?]
at java.nio.Buffer.position(Buffer.java:293) ~[?:?]
at java.nio.ByteBuffer.position(ByteBuffer.java:1094) ~[?:?]
at java.nio.MappedByteBuffer.position(MappedByteBuffer.java:226) ~[?:?]
at java.nio.MappedByteBuffer.position(MappedByteBuffer.java:67) ~[?:?]
at
org.apache.pinot.segment.local.io.writer.impl.VarByteChunkSVForwardIndexWriter.putStrings(VarByteChunkSVForwardIndexWriter.java:119)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at
org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator.putStringMV(MultiValueVarByteRawIndexCreator.java:106)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at
org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.indexRow(SegmentColumnarIndexCreator.java:543)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at
org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:244)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at
org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:131)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:815)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:746)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:644)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
I have added extensive testing for the impacted components and could not
find any buffer overflows, and have narrowed this down to an integer overflow
such that the product of `Integer.BYTES + maxRowLengthInBytes +
maxLengthPrefixes` and a fixed `numDocsPerChunk` overflows and is non-negative.
Very large MV values is a limitation of the current implementation and I have
added precondition checks to detect overflow unambiguously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
