shehabgamin opened a new issue, #27206:
URL: https://github.com/apache/datafusion/issues/27206
### Describe the bug
Window queries can return different results after a physical plan protobuf
round trip. PhysicalWindowExprNode` has no field for some of the state a window
expression carries, so `datafusion-proto` drops it:
- the `FILTER (WHERE ...)` of a window aggregate (only the logical plan side
was fixed, in #17417)
- `IGNORE NULLS` on window functions such as `lag`, `lead`, `first_value`
and `nth_value` (the encoder always writes `false`)
- the reversed flag that the optimizer sets when it evaluates `lag` / `lead`
/ `nth_value` over reversed input
- the ordering and reversed flag of an aggregate used as a window function
(`array_agg(x ORDER BY y) OVER ()` built through the API)
### To Reproduce
```rust
use datafusion::arrow::util::pretty::pretty_format_batches;
use datafusion::physical_plan::collect;
use datafusion::prelude::SessionContext;
use datafusion_proto::bytes::{physical_plan_from_bytes,
physical_plan_to_bytes};
#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
let ctx = SessionContext::new();
let sql = "SELECT a, \
count(*) FILTER (WHERE a > 1) OVER () AS c, \
lag(b) IGNORE NULLS OVER (ORDER BY a DESC) AS l, \
lag(a, 1) OVER (ORDER BY a) AS r \
FROM (VALUES (1, 10), (2, NULL), (3, 30), (4, NULL), (5, 50))
AS t(a, b) \
ORDER BY a DESC";
let plan = ctx.sql(sql).await?.create_physical_plan().await?;
let bytes = physical_plan_to_bytes(plan)?;
let plan = physical_plan_from_bytes(&bytes, &ctx.task_ctx())?;
println!("{}", pretty_format_batches(&collect(plan,
ctx.task_ctx()).await?)?);
Ok(())
}
```
Before the round trip:
```
a | c | l | r
5 | 4 | | 4
4 | 4 | 50 | 3
3 | 4 | 50 | 2
2 | 4 | 30 | 1
1 | 4 | 30 |
```
After the round trip, `c` ignores the filter, `l` stops skipping nulls, and
`r` returns the following row:
```
a | c | l | r
5 | 5 | |
4 | 5 | 50 | 5
3 | 5 | | 4
2 | 5 | 30 | 3
1 | 5 | | 2
```
### Expected behavior
The decoded plan returns the same result as the original plan.
### Additional context
Aggregates in `AggregateExec` (ordering, `DISTINCT`, `IGNORE NULLS`,
`FILTER`, reversal, display names) already round-trip correctly. Only window
expressions are affected.
### Willingness to contribute
I plan to contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
