haohuaijin opened a new issue, #25454: URL: https://github.com/apache/datafusion/issues/25454
### Is your feature request related to a problem or challenge? Integrations can attach a `ParquetAccessPlan` containing a `RowSelection` to a `PartitionedFile`, for example after looking up matching rows in an external index. Queries return the correct rows, but the predicate-based page index loading policy can miss opportunities to skip unselected pages. This occurs when the scan has no page-pruning predicate, or when row-group statistics fully match the scan predicate while an external selection still excludes rows. For example, an external index may handle `service = 'api'`, leaving only a time-range predicate in the Parquet scan. A row group entirely within that time range can be fully matched for the scan predicate while only some rows belong to the requested service. This assumes the service condition is handled externally; matching the time range alone is insufficient if that condition remains in the scan predicate. ### Describe the solution you'd like When page indexes are enabled, also load them if a surviving row group has: - An external selection containing both selected and skipped rows. - An offset index on at least one projected file column. Keep the existing predicate-based loading path as a fallback. Empty, all-selected, and all-skipped selections should not independently trigger the new path, nor should an index on an unprojected column alone. An offset index is sufficient because the selection already identifies the desired rows; column-index min/max statistics are not required for this path. See the [Parquet page index documentation](https://parquet.apache.org/docs/file-format/pageindex/). ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
