prashanthbdremio commented on code in PR #1229:
URL: https://github.com/apache/arrow-java/pull/1229#discussion_r3708636399
##########
vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java:
##########
@@ -263,18 +263,33 @@ public void exportCDataBuffers(List<ArrowBuf> buffers,
ArrowBuf buffersPtr, long
/** Set the reader and writer indexes for the inner buffers. */
private void setReaderAndWriterIndex() {
+ final long requiredOffsetBufferCapacity = (long) (valueCount + 1) *
OFFSET_WIDTH;
validityBuffer.readerIndex(0);
offsetBuffer.readerIndex(0);
if (valueCount == 0) {
validityBuffer.writerIndex(0);
+ ensureEmptyOffsetBufferCapacity(requiredOffsetBufferCapacity);
} else {
validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
}
// IPC serializer will determine readable bytes based on `readerIndex` and
`writerIndex`.
// Both are set to 0 means 0 bytes are written to the IPC stream which
will crash IPC readers
// in other libraries. According to Arrow spec, we should still output the
offset buffer which
// is [0].
- offsetBuffer.writerIndex((long) (valueCount + 1) * OFFSET_WIDTH);
+ offsetBuffer.writerIndex(requiredOffsetBufferCapacity);
+ }
+
+ private void ensureEmptyOffsetBufferCapacity(long requiredCapacity) {
+ if (offsetBuffer.capacity() >= requiredCapacity) {
+ return;
+ }
+ long previousOffsetAllocationSizeInBytes = offsetAllocationSizeInBytes;
+ ArrowBuf oldOffsetBuffer = offsetBuffer;
+ offsetBuffer = allocateOffsetBuffer(requiredCapacity);
+ offsetBuffer.setBytes(
+ 0, oldOffsetBuffer, 0, Math.min(oldOffsetBuffer.capacity(),
requiredCapacity));
Review Comment:
Addressed in the latest push. The helper now copies existing complete offset
entries and fills any newly allocated offset entries with the last copied
offset value, for both `ListVector` and `LargeListVector`.
Validation: `mvn -pl vector -am -P=-error-prone
-Dmaven.gitcommitid.skip=true -Dsurefire.failIfNoSpecifiedTests=false
-Dtest=TestSplitAndTransfer,TestListVector,TestLargeListVector test`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
