mithuncy opened a new pull request, #24124:
URL: https://github.com/apache/datafusion/pull/24124
## Which issue does this PR close?
- Closes #23823.
## Rationale for this change
`LogicalPlanNode::try_from_logical_plan` recursively serializes logical plans
to protobuf. In debug builds, its large `match` compiled to a 202,304 B stack
frame, so ten nested `SubqueryAlias` nodes over an `EmptyRelation` overflowed
a 2 MiB thread stack.
Each arm was isolated independently against the original dispatcher:
| Independent frame effect | Arms | Examples |
| --- | ---: | --- |
| >=10 KiB reduction | 4 | TableScan: 34,608 B; Join: 11,392 B; Dml /
RecursiveQuery: 10,320 B |
| 5-8 KiB reduction | 17 | Projection, Filter, Aggregate, Repartition,
Unnest, Copy |
| 2-4 KiB reduction | 7 | Values, EmptyRelation, Union, Extension |
| <=64 B effect | 9 | Several DDL/statement arms; Subquery adds 16 B |
These effects are not additive: each change alters the compiler's layout of
the same `match` frame. Isolating the 15 arms with the largest independent
reductions still left a 58,560 B frame; isolating all 37 arms reduced it to
1,680 B.
## What changes are included in this PR?
- Isolate every `try_from_logical_plan` match arm behind a debug-only
non-inlined helper, preventing arm-local temporaries from inflating the
recursive dispatcher frame.
- Add the opt-in `datafusion-proto/recursive_protection` feature using the
existing DataFusion recursion pattern.
- Add child-process stack-safety regressions for the original 2 MiB-stack
reproducer and feature-gated stack growth.
With `recursive_protection`, the dispatcher frame measures 1,648 B. The
helper is only forced out of line in debug builds.
## Are these changes tested?
Yes.
- `cargo fmt --all --check`
- `cargo check -p datafusion-proto --all-features`
- `cargo test -p datafusion-proto --lib`
- Stack-safety regression: 100 nested aliases on a 2 MiB stack.
- Feature-gated stack-growth regression: 2,000 nested aliases with
`recursive_protection`.
## Are there any user-facing changes?
`datafusion-proto` gains an opt-in `recursive_protection` feature. Existing
protobuf wire format, conversion behavior, and default features are
unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
