alexanderbianchi opened a new pull request, #26890: URL: https://github.com/apache/datafusion/pull/26890
## Which issue does this PR close? Alternative approach for #26602, presented separately from #26603 for API discussion. ## Rationale for this change Mutable lambda and outer-query stacks on a Substrait consumer prevent safe concurrent consumption on the same consumer. Locks protect individual stack operations, but one branch can still observe another branch's scope between awaits. Error and cancellation paths also require scope cleanup. An explicit immutable context makes scope ownership follow the traversal. Independent branches can share one consumer without cloning it, and nested conversions cannot leave scope state behind on that consumer. ## What changes are included in this PR? - Add `SubstraitConsumerContext` with immutable outer-query and lambda scopes. Entering a scope derives a child context; the parent stays unchanged. - Pass the context through relation and expression callbacks and the corresponding standard conversion helpers, including extension relation handlers. - Remove the consumer's push/pop/query scope methods and `DefaultSubstraitLambdaConsumer`. `DefaultSubstraitConsumer` retains only session and extension configuration. - Consume JOIN and UNION inputs concurrently on the same consumer and context through `consume_rel_inputs`, preserving input order and cancelling pending siblings on failure. - Keep the plan-level entrypoint signatures unchanged; they create root contexts. Document migration for custom consumers and direct helper callers. This draft is an alternative to the compatibility-preserving fork hook in #26603. It does not include that fork API, table caching, materialized CTE encoding, or a change to reference resolution. ## What is the testing strategy for this PR? Tests cover concurrent input ordering and cancellation, concurrent differently typed lambda bodies on one consumer, nested lambda references, error/cancellation isolation, outer-schema ancestry, JOIN/UNION hook dispatch, and result-count validation. Existing integration tests exercise correlated subqueries and lambda semantics. Validation passed on `192bdab6a`: - Formatting and the repository documentation formatter. - Workspace Clippy with all targets and all features, with warnings denied. - 73 Substrait unit tests, 229 integration tests (6 ignored), and 3 doctests. - A real execution comparison before and after round-tripping differently typed sibling lambdas and nested captures. Three plan snapshots were updated for lexical parameter-name reuse; nested bindings and query results are preserved. ## Are there any user-facing changes? **Breaking API change:** custom relation/expression callbacks and direct conversion-helper calls must accept and forward `&SubstraitConsumerContext`. Scope management moves from the consumer to the context. The Substrait README includes migration instructions. Default input consumption becomes concurrent. Custom consumers with other per-traversal state must adapt it or override `consume_rel_inputs` to preserve sequential scheduling. Shared services and caches must support concurrent callbacks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
