rdblue commented on code in PR #6417:
URL: https://github.com/apache/iceberg/pull/6417#discussion_r1051684313
##########
parquet/src/main/java/org/apache/iceberg/parquet/ReadConf.java:
##########
@@ -185,21 +184,16 @@ private Map<Long, Long> generateOffsetToStartPos(Schema
schema) {
return null;
}
- try (ParquetFileReader fileReader = newReader(file,
ParquetReadOptions.builder().build())) {
Review Comment:
Unfortunately, this isn't correct. The options that are passed into ReadConf
will include the split's range of bytes in the file, and Parquet will use that
to filter metadata before returning it, so `fileReader.getRowGroups()` will be
different. We need to avoid passing down that filter.
I think it's debatable whether we need to actually filter metadata. It seems
like a silly optimization to me, when we will generally have very few row
groups in a file. I think it was an optimization for Parquet doing some pretty
dumb things in the past.
If you want, we can [remove the range
pushdown](https://github.com/apache/iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L1027)
and continue with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
