gaborkaszab commented on PR #16583: URL: https://github.com/apache/iceberg/pull/16583#issuecomment-4768516532
Thanks for the PR @kamcheungting-db ! When I was working on the `PartitionStatisticsScan` API I was also thinking about adding a filter to the internal readers. I also observed that for Parquet this might make sense for row group skipping and it has no filtering effect on other file formats. I have 2 questions wrt this: 1) **Size of a row in the PartitionStatistics Parquet file** is 76bytes + the size of the partition struct. Depends on what we use as partition field (could be 4 byte per numeric field like bucket or date time related ones, or the length of a string for the string columns), but we can calculate with 100-200 byte per row in the file. So going with this size and the default 128 MB Parquet row group size, it takes 640k partition to have more than a single row group in the `PartitionStatistics` file. I'm not sure how realistic is that in real life. So when I was working on this area I figured that in practice there is not much benefit in having row group filtering for these stat files. What I also had in mind is that I'll come back revisit this calculation when we extend the partition stats for instance with column stats (min, max, etc. that we have in `ContentStats` class) but this is further down the road so I saw no point implementing now. Would you mind sharing some real world benefits for this, @kamcheungting-db? 3) Implementation-wise, **instead of introducing another variation of `InternalData.read()`** function, would it make sense to extend `InternalData.ReadBuilder` interface with a `filter(Expression)` only? You could use it like this: `InternalData.read(fileFormat, inputFile).project(...).setRootType(...).filter(...).build()` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
