parthchandra commented on code in PR #999:
URL: https://github.com/apache/parquet-mr/pull/999#discussion_r997548363
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1093,10 +1099,38 @@ private ColumnChunkPageReadStore
internalReadFilteredRowGroup(BlockMetaData bloc
}
}
}
- // actually read all the chunks
+ // Vectored IO up.
+
+ List<FileRange> ranges = new ArrayList<>();
for (ConsecutivePartList consecutiveChunks : allParts) {
- consecutiveChunks.readAll(f, builder);
+ ranges.add(FileRange.createFileRange(consecutiveChunks.offset, (int)
consecutiveChunks.length));
+ }
+ LOG.warn("Doing vectored IO for ranges {}", ranges);
+ f.readVectored(ranges, ByteBuffer::allocate);
Review Comment:
> Don't you think it is similar to what I have done in the commented
readVectored() implementation?
Yes, it is. However, the parallel reading will split a large range as
opposed to what the vectored read will do.
> I guess we need to run some benchmarks. First, I need to see what type of
ranges parquet generates for real-world/tpcds queries. Do you have that by any
chance?
No I don't have the ranges. TPCDS is a large set of queries. Pick a scale
factor and a couple of queries and I'll see what I can get for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
