crepererum opened a new issue, #3968:
URL: https://github.com/apache/arrow-datafusion/issues/3968
**Describe the bug**
`Expr::to_bytes` with deeply nested expressions can produce a byte sequence
that cannot be read by `Expr::from_bytes` because the prost recursion limit is
hit. This is suboptimal in distributed application because it would be good if
a client can decide if an expression is wire/roundtrip-safe and potentially
adjust its predicate pushdown accordingly.
**To Reproduce**
```rust
#[test]
fn roundtrip_deeply_nested() {
// we need more stack space so this doesn't overflow in dev builds
std::thread::Builder::new().stack_size(10_000_000).spawn(|| {
// don't know what "too much" is, so let's slowly try to increase
complexity
let n_max = 100;
for n in 1..n_max {
println!("testing: {n}");
let expr_base = col("a").lt(lit(5i32));
let expr = (0..n).fold(expr_base.clone(), |expr, _|
expr.and(expr_base.clone()));
// Convert it to an opaque form
let bytes = match expr.to_bytes() {
Ok(bytes) => bytes,
Err(_) => {
// found expression that is too deeply nested
return;
}
};
// Decode bytes from somewhere (over network, etc.
let decoded_expr = Expr::from_bytes(&bytes).expect("serialization
worked, so deserialization should work as well");
assert_eq!(expr, decoded_expr);
}
panic!("did not find a 'too deeply nested' expression, tested up to a
depth of {n_max}")
}).expect("spawning thread").join().expect("joining thread");
}
```
**Expected behavior**
Above test passes.
**Additional context**
Above test fails with:
`thread '<unnamed>' panicked at 'serialization worked, so deserialization
should work as well: Plan("Error decoding expr as protobuf: failed to decode
Protobuf message: LogicalExprNode.expr_type: BinaryExprNode.l:
LogicalExprNode.expr_type: .....recursion limit reached")'`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
