yihua opened a new issue, #923: URL: https://github.com/apache/hudi-rs/issues/923
### Feature Description Filter pushdown into the parquet reader (#911) prunes whole row groups by footer statistics and evaluates a row filter inside the row groups that remain, but it never uses the page index. The base file reader loads only the footer, so a row group with any matching row still has every projected column chunk fetched and decompressed in full, and pages whose column index rules the filter out are read anyway. A selective filter on a large row group (the 120 MB default) therefore pays for the whole group. Proposed: load the page index when a read carries filters, turn the column index of the filter columns into a `RowSelection` under the same gates #911 applies (matching column types, record key filters only when log files merge, no range pruning on float statistics), and let the offset index limit fetches to the selected pages. Files without a page index keep today's behavior. The post-merge row mask stays as is. ### Additional context Spark and parquet-mr write the column and offset indexes by default, as does the arrow-rs writer, so existing tables benefit without a rewrite. Loading the index costs one more range read per file, which is why it should only happen for filtered reads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
