Fokko commented on code in PR #3620:
URL: https://github.com/apache/parquet-java/pull/3620#discussion_r3448898610
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1361,8 +1362,32 @@ private void readVectored(List<ConsecutivePartList>
allParts, ChunkListBuilder b
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges",
totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
+ // Wrap the allocator to track buffers allocated during the vectored read.
+ // Hadoop's vectored IO may merge ranges and return slices of merged
buffers,
+ // so the buffers returned via CompletableFuture may differ from those
originally
+ // allocated. We track the originals here to ensure they are properly
released.
+ ByteBufferAllocator baseAllocator = options.getAllocator();
+ List<ByteBuffer> allocatedBuffers = new ArrayList<>();
+ ByteBufferAllocator trackingAllocator = new ByteBufferAllocator() {
Review Comment:
I don't like adding a wrapper here just to fix something in Hadoop.
@steveloughran suggested to set the config to avoid checksums on the Hadoop
side:
https://github.com/apache/parquet-java/issues/3356#issuecomment-4089172411 This
doesn't add any value to Parquet anyway. This is more or less what I suggested
in https://github.com/apache/parquet-java/pull/3559/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
