wirybeaver opened a new pull request, #2749: URL: https://github.com/apache/datafusion-ballista/pull/2749
## Motivation Part of #2700 (Task 2). Build-side information must reach probe tasks before their scans and shuffles can benefit from it. This second layer builds on the [runtime filter operators and task reports](https://github.com/apache/datafusion-ballista/pull/2701) and delivers opt-in cross-stage row filtering through AQE. ## Modification Attach filter taps only to executable stage copies after AQE chooses existing boundaries. Keep the AQE tree and ordinary stage selection unchanged, and bind consumers to the final build key. A job-scoped hub uses reports from the existing task-status lifecycle so lost and retried attempts cannot leave stale summaries. ```text build task reports -> job-scoped hub -> complete producer coverage | probe stage ready -------------------------------+ v freeze complete filters / bypass -> dispatch ``` Probe stages proceed immediately. Only complete producer coverage can supply a filter; unavailable reports bypass, and the stage plan is frozen before its first dispatch. Saturated Blooms skip further hashing while NULL and MinMax collection continue. Candidate selection uses AQE's size comparison and targets scan paths where filtering can save predicate or shuffle work. The feature is disabled by default and covers the first supported direct-column equality key of inner and semi hash joins, including NULL-safe equality. ## Test A standalone scheduler/executor test uses a Parquet build with known cardinality and a CSV probe to verify actual pruning after the existing build-stage dependency completes. It compares filtered and unfiltered results for ordinary and NULL-safe equality, an outer join and a limited probe subquery, and checks independently runnable partitioned stages preserve results when no report is ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
