joyhaldar commented on code in PR #17610:
URL: https://github.com/apache/iceberg/pull/17610#discussion_r3885661659
##########
data/src/test/java/org/apache/iceberg/data/ReadFormatModelTests.java:
##########
@@ -543,7 +544,7 @@ void testReaderBuilderSplit(FileFormat fileFormat) throws
IOException {
@ParameterizedTest
@FieldSource("FILE_FORMATS")
void testReaderBuilderReuseContainers(FileFormat fileFormat) throws
IOException {
-
+ assumeFalse(supportsBatchReads(), "Batch reads reuse differently");
Review Comment:
Neither vectorized reader returns the same object twice.
[ColumnarBatchReader.read](https://github.com/apache/iceberg/blob/e098577b287adfdb526822265aefb5addde0f3fe/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchReader.java#L51)
always builds a new `ColumnarBatch` and uses reuse only to decide whether to
close the previous vectors IIUC:
```java
if (reuse == null) {
closeVectors();
}
return new ColumnBatchLoader(numRowsToRead).loadDataToColumnBatch();
```
[VectorizedSparkOrcReaders.read](https://github.com/apache/iceberg/blob/e098577b287adfdb526822265aefb5addde0f3fe/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkOrcReaders.java#L60)
doesn't take a reuse argument at all.
I checked with and without `reuseContainers()` on both formats and each call
returns a new object either way. Is that expected for batch reads? Please
correct me if I am wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
