haohuaijin opened a new pull request, #25460: URL: https://github.com/apache/datafusion/pull/25460
## Which issue does this PR close? Closes #25454. ## Rationale for this change External row selections can benefit from offset indexes without a page-pruning predicate, or when row-group statistics fully match the scan predicate. For example, an external index can select rows for a service while the Parquet scan checks only a time range. A row group entirely within that range is fully matched for the predicate, but the external selection can still exclude most rows. Offset indexes allow the reader to locate relevant pages by row position. Column-index statistics are unnecessary for this path because the selection already identifies the rows to read. ## What changes are included in this PR? - Explicitly honor `enable_page_index` before either loading path. - Load indexes when a surviving row group has a partial external selection and an offset index on a projected Parquet leaf column. - Detect partial selections using a second alternating run instead of counting all selected and skipped rows. Bitmap selections use a streaming iterator without materializing selectors. - Resolve the projection with the existing read-plan logic, handling nested fields and excluding virtual columns. - Keep the existing predicate-based fallback, including its handling of predicate columns outside the output projection. The projection check controls whether loading is triggered; it does not restrict index I/O to projected columns. Read-plan caching, decoder refactoring, and arrow-rs changes are outside this PR. ## What is the testing strategy for this PR? Unit tests cover offset-only metadata, missing indexes, fully matched and skipped row groups, selector/bitmap representations, empty/uniform selections, and indexes only on unprojected columns. They also verify the predicate-based fallback. A V1 Parquet test selects the final 100 of 10,000 rows and checks exact output values and reduced `bytes_scanned`, both without a predicate and with a fully matching predicate. The following checks passed, including all 11 page-index tests: ```bash cargo test -p datafusion-datasource-parquet --lib page_index cargo clippy -p datafusion-datasource-parquet --all-targets --all-features -- -D warnings cargo fmt --all -- --check ``` The repository-wide `./dev/rust_lint.sh` could not proceed because the local Python environment is missing PyYAML. The targeted checks above passed. ## Are there any user-facing changes? External partial row selections can use offset indexes to skip unselected pages without a useful page-pruning predicate. Query results and public APIs are unchanged, and disabling page indexes still disables this loading path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
