gaborkaszab commented on code in PR #16582:
URL: https://github.com/apache/iceberg/pull/16582#discussion_r3324437964
##########
core/src/main/java/org/apache/iceberg/BasePartitionStatisticsScan.java:
##########
@@ -71,16 +91,46 @@ public CloseableIterable<PartitionStatistics> scan() {
return CloseableIterable.empty();
}
+ // Always read with the latest schema
Types.StructType partitionType = Partitioning.partitionType(table);
- Schema schema = PartitionStatistics.schema(partitionType,
TableUtil.formatVersion(table));
+ Schema schema = PartitionStatistics.latestSchema(partitionType);
+ Schema readSchema = readSchema(schema);
FileFormat fileFormat = FileFormat.fromFileName(statsFile.get().path());
Preconditions.checkNotNull(
fileFormat != null, "Unable to determine format of file: %s",
statsFile.get().path());
- return InternalData.read(fileFormat,
table.io().newInputFile(statsFile.get().path()))
- .project(schema)
- .setRootType(BasePartitionStatistics.class)
- .build();
+ CloseableIterable<PartitionStatistics> result =
+ InternalData.read(fileFormat,
table.io().newInputFile(statsFile.get().path()))
+ .project(readSchema)
Review Comment:
Thanks for bringing this up, @kamcheungting-db!
I explored this as well, apparently not all of the file formats have a
filter method. But even if they had, I observed that Parquet uses the filter
only to skip row groups but it won't filter the rows after row group filtering.
For stats files row group filtering doesn't seem very useful, probable all the
stats fit into the same row group with the current content.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
