iemejia commented on code in PR #3579:
URL: https://github.com/apache/parquet-java/pull/3579#discussion_r3602662529
##########
pom.xml:
##########
@@ -80,10 +80,10 @@
<jackson-annotations.version>2.21</jackson-annotations.version>
<japicmp.version>0.25.7</japicmp.version>
<javax.annotation.version>1.3.2</javax.annotation.version>
- <spotless.version>2.46.1</spotless.version>
+ <spotless.version>3.5.1</spotless.version>
<shade.prefix>shaded.parquet</shade.prefix>
<!-- Guarantees no newer classes/methods/constants are used by parquet. -->
- <hadoop.version>3.3.0</hadoop.version>
+ <hadoop.version>3.4.3</hadoop.version>
Review Comment:
Good call. I dropped the Hadoop bump from this PR — I verified the Java 17
build (compile + tests) passes fine on Hadoop 3.3.0 (vectored-I/O tests just
skip, no buffer leaks), so it isn't needed here. Let's have the bump land via
#3559.
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1361,12 +1362,42 @@ private void readVectored(List<ConsecutivePartList>
allParts, ChunkListBuilder b
totalSize += len;
}
LOG.debug("Reading {} bytes of data with vectored IO in {} ranges",
totalSize, ranges.size());
- // Request a vectored read;
- f.readVectored(ranges, options.getAllocator());
- int k = 0;
- for (ConsecutivePartList consecutivePart : allParts) {
- ParquetFileRange currRange = ranges.get(k++);
- consecutivePart.readFromVectoredRange(currRange, builder);
+ // Use a capturing allocator to track all buffers allocated by Hadoop
during vectored reads.
Review Comment:
Removed from this PR as well — it's only relevant once vectored I/O is
actually exercised (Hadoop 3.4.x). Deferring to your approach in #3559.
##########
pom.xml:
##########
@@ -80,10 +80,10 @@
<jackson-annotations.version>2.21</jackson-annotations.version>
<japicmp.version>0.25.7</japicmp.version>
<javax.annotation.version>1.3.2</javax.annotation.version>
- <spotless.version>2.46.1</spotless.version>
+ <spotless.version>3.5.1</spotless.version>
Review Comment:
Done — bumped to `3.8.0` and ran `spotless:apply` (no formatting changes
beyond the version bump). Thanks for the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
