joyhaldar commented on code in PR #17610:
URL: https://github.com/apache/iceberg/pull/17610#discussion_r4305743322
##########
data/src/test/java/org/apache/iceberg/data/ReadFormatModelTests.java:
##########
@@ -488,6 +488,7 @@ void testReaderBuilderCaseSensitive(FileFormat fileFormat)
throws IOException {
@FieldSource("FILE_FORMATS")
void testReaderBuilderSplit(FileFormat fileFormat) throws IOException {
+ assumeFalse(supportsBatchReads(), "Batch reads return batches, not rows");
Review Comment:
I added a `countRows()` hook that sums the row counts across `ColumnarBatch`
instances and uses `.size()` for row readers. This allows the shared tests to
work for both.
##########
data/src/test/java/org/apache/iceberg/data/ReadFormatModelTests.java:
##########
@@ -543,7 +544,7 @@ void testReaderBuilderSplit(FileFormat fileFormat) throws
IOException {
@ParameterizedTest
@FieldSource("FILE_FORMATS")
void testReaderBuilderReuseContainers(FileFormat fileFormat) throws
IOException {
-
+ assumeFalse(supportsBatchReads(), "Batch reads reuse differently");
Review Comment:
The reuse logic is in `VectorizedArrowReader.java`:
https://github.com/apache/iceberg/blob/main/arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java#L140-L156
In `ColumnarBatchReader.java`, the previous VectorHolder is passed back to
the reader, allowing the underlying Arrow vector to be reused.
https://github.com/apache/iceberg/blob/524fe9ef1b4aa72be19623f667738b21a02b23b2/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java#L81
I am getting each vector in a batch like this,
```java
private ValueVector arrowVector(ColumnarBatch batch, int col) {
IcebergArrowColumnVector vector = (IcebergArrowColumnVector)
batch.column(col);
return vector.vectorAccessor().getVector();
}
```
I have updated the shared test to compare the underlying Arrow vectors for
every column, both with and without `reuseContainers()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
