urvishdesai commented on code in PR #41967:
URL: https://github.com/apache/arrow/pull/41967#discussion_r1631736490
##########
java/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
@@ -1667,4 +1667,23 @@ protected void getData(int index, ReusableBuffer<?>
buffer) {
public <OUT, IN> OUT accept(VectorVisitor<OUT, IN> visitor, IN value) {
return visitor.visit(this, value);
}
+
+ @Override
+ public void exportCDataBuffers(List<ArrowBuf> buffers, ArrowBuf buffersPtr,
long nullValue) {
+ exportBuffer(validityBuffer, buffers, buffersPtr, nullValue, true);
+
+ if (viewBuffer.capacity() == 0) {
+ // Empty view buffer is allowed here.
+ // We set `retain = false` to explicitly not increase the ref count for
the exported buffer.
+ // The ref count of the newly created buffer (i.e., 1) already
represents the usage
+ // of the imported side.
+ exportBuffer(allocator.getEmpty(), buffers, buffersPtr, nullValue,
false);
+ } else {
+ exportBuffer(viewBuffer, buffers, buffersPtr, nullValue, true);
+ }
+
+ for (int i = 0; i < dataBuffers.size(); i++) {
+ exportBuffer(dataBuffers.get(i), buffers, buffersPtr, nullValue, true);
+ }
Review Comment:
We need another buffer to be appended at the end that stores the sizes of
each of the data buffer. This buffer from the spec is missing here. This is
probably why you are seeing "Expected at least 3 buffers for imported type"
error.
##########
java/c/src/main/java/org/apache/arrow/c/BufferImportTypeVisitor.java:
##########
@@ -210,9 +223,48 @@ public List<ArrowBuf> visit(ArrowType.Utf8 type) {
}
}
+ private List<ArrowBuf> visitVariableWidthView(ArrowType type) {
+ try (ArrowBuf view = importView(type)) {
+ List<ArrowBuf> buffers = new ArrayList<>();
+ view.getReferenceManager().retain();
+ ArrowBuf maybeValidityBuffer = maybeImportBitmap(type);
+ buffers.add(maybeValidityBuffer);
+ buffers.add(view);
+ final int elementSize = BaseVariableWidthViewVector.ELEMENT_SIZE;
+ final int lengthWidth = BaseVariableWidthViewVector.LENGTH_WIDTH;
+ final int prefixWidth = BaseVariableWidthViewVector.PREFIX_WIDTH;
+ final int lengthPrefixWidth = lengthWidth + prefixWidth;
+ // Map to store the data buffer index and the total length of data in
that buffer
+ Map<Integer, Long> dataBufferInfo = new HashMap<>();
+ for (int i = 0; i < fieldNode.getLength(); i++) {
+ final int length = view.getInt((long) i * elementSize);
+ if (length > BaseVariableWidthViewVector.INLINE_SIZE) {
+ checkState(maybeValidityBuffer != null,
+ "Validity buffer is required for data of type " + type);
+ if (BitVectorHelper.get(maybeValidityBuffer, i) == 1) {
+ final int bufferIndex =
+ view.getInt(((long) i * elementSize) + lengthPrefixWidth);
+ if (dataBufferInfo.containsKey(bufferIndex)) {
+ dataBufferInfo.compute(bufferIndex, (key, value) -> value !=
null ? value + (long) length : 0);
+ } else {
+ dataBufferInfo.put(bufferIndex, (long) length);
+ }
+ }
+ }
+ }
Review Comment:
The last buffer was added to the arrow spec to store the length of each of
the data buffers. You can use the values of this buffer as array of lengths to
import the variably sized data buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
