alexanderbianchi opened a new pull request, #26896: URL: https://github.com/apache/datafusion/pull/26896
## Which issue does this PR close? Part of #26602. Stacked on #26603; merge the input hook first. Both PRs target Apache `main`, so this draft includes the prerequisite commit. [Fork-only diff](https://github.com/alexanderbianchi/datafusion/compare/codex/substrait-concurrent-consumer...codex/substrait-consumer-forks). ## Rationale for this change Concurrent inputs cannot safely share mutable lambda and outer-query scope stacks. Consumers need a way to give each branch its own scope. ## What changes are included in this PR? Add optional `fork_for_branch() -> Option<Self>`. `consume_rel_inputs` consumes isolated forks concurrently when all inputs can be forked, otherwise it stays sequential. The default consumer implements forking by copying scope stacks and sharing session configuration. This extracts the fork approach previously included in #26603. Table caching and lookup limits remain separate in #26680; #26890 explores explicit contexts instead. ## What is the testing strategy for this PR? Existing tests cover sequential fallback, concurrent completion in input order, cancellation on error, and inherited but independent scopes. This is the same fork code already validated in #26603: formatting, workspace Clippy, 73 Substrait unit tests, 228 integration tests, 3 doctests, and the extended workspace suite passed. ## Are there any user-facing changes? The default consumer consumes JOIN/UNION inputs concurrently. Custom consumers remain sequential unless they implement the fork hook or override `consume_rel_inputs`. Existing implementations remain source-compatible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
