joyhaldar commented on code in PR #17610:
URL: https://github.com/apache/iceberg/pull/17610#discussion_r4305743661
##########
data/src/test/java/org/apache/iceberg/data/ReadFormatModelTests.java:
##########
@@ -543,7 +544,7 @@ void testReaderBuilderSplit(FileFormat fileFormat) throws
IOException {
@ParameterizedTest
@FieldSource("FILE_FORMATS")
void testReaderBuilderReuseContainers(FileFormat fileFormat) throws
IOException {
-
+ assumeFalse(supportsBatchReads(), "Batch reads reuse differently");
Review Comment:
The reuse logic is in `VectorizedArrowReader.java`.
https://github.com/apache/iceberg/blob/main/arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java#L140-L156
In `ColumnarBatchReader.java`, the previous VectorHolder is passed back to
the reader, allowing the underlying Arrow vector to be reused.
https://github.com/apache/iceberg/blob/524fe9ef1b4aa72be19623f667738b21a02b23b2/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java#L81
I am getting each vector in a batch like this,
```java
private ValueVector arrowVector(ColumnarBatch batch, int col) {
IcebergArrowColumnVector vector = (IcebergArrowColumnVector)
batch.column(col);
return vector.vectorAccessor().getVector();
}
```
I have updated the shared test to compare the underlying Arrow vectors for
every column, both with and without `reuseContainers()`.
##########
data/src/test/java/org/apache/iceberg/data/ReadFormatModelTests.java:
##########
@@ -1343,7 +1347,7 @@ void testReadMetadataColumnRowPosition(FileFormat
fileFormat) throws IOException
@ParameterizedTest
@FieldSource("FILE_FORMATS")
void testReadMetadataColumnIsDeleted(FileFormat fileFormat) throws
IOException {
-
+ assumeFalse(supportsBatchReads(), "DeletedColumnVector requires a delete
filter");
Review Comment:
I updated the skip message to explain the issue and limited the skip to
vectorized Parquet. ORC still runs the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
