sajjad-moradi opened a new pull request, #13717:
URL: https://github.com/apache/pinot/pull/13717
Memory allocation for FixedByteSVMutableForwardIndex with large segment size
throws the following exception because of int overflow:
```
Caused by: java.lang.IllegalArgumentException: Illegal memory allocation -4
for segment TestTable__2__18__20240727T1658Z column
TestTable__2__18__20240727T1658Z:action.sv.unsorted.fwd
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
~[com.google.guava.guava-33.1.0-jre.jar:?]
at
org.apache.pinot.segment.local.io.readerwriter.RealtimeIndexOffHeapMemoryManager.allocate(RealtimeIndexOffHeapMemoryManager.java:78)
~[org.apache.pinot.pinot-segment-local-1.2.0-dev-1372.jar:1.2.0-dev-1372-d1cf4859df5749da8115e00b5055f47185b8c4b2]
at
org.apache.pinot.segment.local.realtime.impl.forward.FixedByteSVMutableForwardIndex.addBuffer(FixedByteSVMutableForwardIndex.java:244)
~[org.apache.pinot.pinot-segment-local-1.2.0-dev-1372.jar:1.2.0-dev-1372-d1cf4859df5749da8115e00b5055f47185b8c4b2]
at
org.apache.pinot.segment.local.realtime.impl.forward.FixedByteSVMutableForwardIndex.<init>(FixedByteSVMutableForwardIndex.java:87)
~[org.apache.pinot.pinot-segment-local-1.2.0-dev-1372.jar:1.2.0-dev-1372-d1cf4859df5749da8115e00b5055f47185b8c4b2]
at
org.apache.pinot.segment.local.realtime.impl.forward.FixedByteSVMutableForwardIndex.<init>(FixedByteSVMutableForwardIndex.java:92)
~[org.apache.pinot.pinot-segment-local-1.2.0-dev-1372.jar:1.2.0-dev-1372-d1cf4859df5749da8115e00b5055f47185b8c4b2]
at
org.apache.pinot.segment.local.segment.index.forward.ForwardIndexType.createMutableIndex(ForwardIndexType.java:309)
~[org.apache.pinot.pinot-segment-local-1.2.0-dev-1372.jar:1.2.0-dev-1372-d1cf4859df5749da8115e00b5055f47185b8c4b2]
at
org.apache.pinot.segment.local.segment.index.forward.ForwardIndexType.createMutableIndex(ForwardIndexType.java:67)
~[org.apache.pinot.pinot-segment-local-1.2.0-dev-1372.jar:1.2.0-dev-1372-d1cf4859df5749da8115e00b5055f47185b8c4b2]
at
org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.addMutableIndex(MutableSegmentImpl.java:412)
~[org.apache.pinot.pinot-segment-local-1.2.0-dev-1372.jar:1.2.0-dev-1372-d1cf4859df5749da8115e00b5055f47185b8c4b2]
at
org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:346)
~[org.apache.pinot.pinot-segment-local-1.2.0-dev-1372.jar:1.2.0-dev-1372-d1cf4859df5749da8115e00b5055f47185b8c4b2]
at
org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.<init>(RealtimeSegmentDataManager.java:1614)
~[org.apache.pinot.pinot-core-1.2.0-dev-1372.jar:1.2.0-dev-1372-d1cf4859df5749da8115e00b5055f47185b8c4b2]
at
org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.doAddConsumingSegment(RealtimeTableDataManager.java:494)
~[org.apache.pinot.pinot-core-1.2.0-dev-1372.jar:1.2.0-dev-1372-d1cf4859df5749da8115e00b5055f47185b8c4b2]
```
-4 is because of int overflow for multiplication of `numRowsPerChunk *
_valueSizeInBytes (2^31-1 * 4)` in which both variables are int.
This PR uses a proper error message for easier debugging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
