RussellSpitzer commented on code in PR #17742:
URL: https://github.com/apache/iceberg/pull/17742#discussion_r3906688287
##########
arrow/src/test/java/org/apache/iceberg/arrow/vectorized/TestArrowReader.java:
##########
@@ -389,6 +391,43 @@ public void testTimestampMillisAreReadCorrectly() throws
Exception {
assertThat(totalRowsRead).as("Should read all
rows").isEqualTo(millisValues.size());
}
+ @Test
+ public void testReleasesDecodedVectorsWhenDictEncodedBatchIsMaterialized()
throws Exception {
+ Table table = createDictEncodedTable();
+ long allocatedBefore =
ArrowAllocation.rootAllocator().getAllocatedMemory();
Review Comment:
So this is a little unsafe for us to do. I know this is probably done a
bunch in this test but we want to isolate the allocators we are using within
the test. Otherwise we have the potential of future concurrent tests on the
same JVM corrupting this test and causing it to fail unecessarily.
I think the best route here is to try to get our test to work off a
different allocator. So we probably need to add a another constructor
```java
public VectorizedReaderBuilder(
Schema expectedSchema,
MessageType parquetSchema,
boolean setArrowValidityVector,
Map<Integer, ?> idToConstant,
Function<List<VectorizedReader<?>>, VectorizedReader<?>> readerFactory,
BufferAllocator bufferAllocator) {
this(
expectedSchema,
parquetSchema,
setArrowValidityVector,
idToConstant,
readerFactory,
(type, value) -> value,
bufferAllocator);
}
```
Then this test can be completely clean and jvm memory safe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
