mithuncy opened a new issue, #23823:
URL: https://github.com/apache/datafusion/issues/23823
`datafusion_proto::bytes::logical_plan_to_bytes` can abort with a native
stack overflow in a debug build while serializing a small, valid `LogicalPlan`.
The overflow occurs during recursive logical-plan-to-protobuf conversion in:
```rust
LogicalPlanNode::try_from_logical_plan(...)
```
## Reproducer
Tested on macOS/aarch64 with DataFusion 54.0.0.
```rust
use std::sync::Arc;
use datafusion_common::DFSchema;
use datafusion_expr::logical_plan::{EmptyRelation, LogicalPlan,
LogicalPlanBuilder};
use datafusion_proto::bytes::logical_plan_to_bytes;
fn main() {
let mut plan = LogicalPlan::EmptyRelation(EmptyRelation {
produce_one_row: false,
schema: Arc::new(DFSchema::empty()),
});
// SubqueryAlias x 10 -> EmptyRelation.
// This requires 11 try_from_logical_plan invocations during
serialization.
for level in 0..10 {
plan = LogicalPlanBuilder::from(plan)
.alias(format!("level_{level}"))
.unwrap()
.build()
.unwrap();
}
std::thread::Builder::new()
.name("two-megabyte-stack".into())
.stack_size(2 * 1024 * 1024)
.spawn(move || logical_plan_to_bytes(&plan).unwrap())
.unwrap()
.join()
.unwrap();
}
```
The default debug profile aborts:
```text
thread 'two-megabyte-stack' has overflowed its stack
fatal runtime error: stack overflow, aborting
```
The same reproducer completes with `cargo run --release`.
## Measurement
I added a temporary AArch64 stack-pointer probe at entry to
`LogicalPlanNode::try_from_logical_plan`.
In the instrumented debug build, each nested entry adds approximately 194 KB
of stack:
```text
depth 2: 193,952 bytes
depth 3: 387,936 bytes
depth 10: 1,745,824 bytes
depth 11: 1,939,808 bytes
depth 12: 2,133,792 bytes
```
The 2 MiB test thread aborts while attempting to enter depth 11. With a 4
MiB test thread, depth 11 completes and the 11-alias plan reaches depth 12.
In the identically instrumented release build, each nested entry adds 14,976
bytes; depth 12 completes at 164,736 bytes.
The exact sizes are target/toolchain dependent, but the large
debug-versus-release difference is reproducible.
## Expected behavior
Serializing this valid logical plan should complete, or return a recoverable
error. It should not abort the process through native stack overflow.
## Possible direction
`try_from_logical_plan` is a large recursive `match` over all `LogicalPlan`
variants. Reducing the size of the recursive path, or using iterative
post-order conversion, would make logical-plan serialization stack-safe.
Related prior art:
- #1047 reduced a large recursive debug stack frame by separating
non-recursive work from the recursive path.
- #5693 tracks stack-safe logical-expression serde.
- #16788 tracks broader DataFusion stack-safety restructuring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
