ShubhamChaurasia commented on a change in pull request #706: HIVE-21966: Llap
external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
URL: https://github.com/apache/hive/pull/706#discussion_r301059071
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java
##########
@@ -365,27 +369,74 @@ private void writeStruct(NonNullableStructVector
arrowVector, StructColumnVector
}
}
- private void writeList(ListVector arrowVector, ListColumnVector hiveVector,
ListTypeInfo typeInfo, int size,
+ // selected[] points to the valid/filtered/selected records at row level.
+ // for MultiValuedColumnVector such as ListColumnVector one record of
vector points to multiple nested records.
+ // In child vectors we get these records in exploded manner i.e. the
number of records in child vectors can have size more
+ // than actual the VectorizedRowBatch, consequently selected[] also needs
to be readjusted.
+ // This method creates a shallow copy of VectorizedRowBatch with corrected
size and selected[]
+
+ private static VectorizedRowBatch
correctSelectedAndSize(VectorizedRowBatch sourceVrb,
+ ListColumnVector
listColumnVector) {
+
+ VectorizedRowBatch vrb = new VectorizedRowBatch(sourceVrb.numCols,
sourceVrb.size);
+ vrb.cols = sourceVrb.cols;
+ vrb.endOfFile = sourceVrb.endOfFile;
+ vrb.projectedColumns = sourceVrb.projectedColumns;
+ vrb.projectionSize = sourceVrb.projectionSize;
+ vrb.selectedInUse = sourceVrb.selectedInUse;
+
+ int correctedSize = 0;
+ final int[] srcVrbSelected = sourceVrb.selected;
+ for (int i = 0; i < sourceVrb.size; i++) {
+ correctedSize += listColumnVector.lengths[srcVrbSelected[i]];
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
