andygrove opened a new pull request, #2558: URL: https://github.com/apache/datafusion-ballista/pull/2558
# Which issue does this PR close? Closes #2551. # Rationale for this change DataFusion builds a file scan's file groups before any predicate is applied: files are sorted by path and cut into contiguous runs. Pruning from Parquet statistics only happens later, inside each task. When the files are clustered by the filtered column (for example Hive-style date directories filtered by a date range), every file that survives pruning sits in a few neighbouring groups. Only the tasks that own those groups have rows to read, the other tasks open their files and skip them, and the stage takes as long as the busiest few tasks. # What changes are included in this PR? - New `BalanceFileGroups` physical optimizer rule (`ballista/scheduler/src/physical_optimizer/file_group_balance.rs`). It evaluates the scan's predicate against the file statistics collected during planning, drops files that cannot match, and deals the remaining files round-robin over the scan's file groups. - The rule leaves a scan untouched when it declares an output partitioning or ordering, must preserve file order, has a single group, has no useful predicate, or when nothing can be pruned. Files without statistics are always kept. The group count never grows, and shrinks only when fewer files survive than there were groups. - It is applied in the static planner right after the physical plan is created, and in AQE plan preparation right after `FilterPushdown` (so the predicate is on the scan and DataFusion's file scan repartitioning sees the pruned files). Physical plans submitted directly by a client are not rewritten. - New setting `ballista.optimizer.balance_scan_file_groups` (default `true`) to turn it off, with the generated `configs.md` updated. Tests: unit tests for the rule, including a case that plans a real `ListingTable` over Parquet files clustered by the filtered column. # Are there any user-facing changes? A new config option, `ballista.optimizer.balance_scan_file_groups`. Scans with selective predicates on clustered columns may now plan fewer, better balanced file groups. No public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
