paleolimbot opened a new issue, #1232:
URL: https://github.com/apache/sedona-db/issues/1232
## Problem
SedonaDB currently depends on DataFusion 54.1.0, which contains a logical
optimizer correctness bug in `PushDownLeafProjections` when nested-field access
is applied to a struct produced by `Unnest`.
For a plan shaped like:
```text
Projection: get_field(dump, "geom")
Unnest: dump
Projection: ST_Dump(geometry) AS dump
```
the optimizer can push `get_field(dump, "geom")` below `Unnest`. At that
point `dump` is still `List<Struct<...>>`, rather than the post-unnest
`Struct<...>`, and optimization fails with:
```text
Optimizer rule 'push_down_leaf_projections' failed
caused by
Execution error: Cannot access field at argument 1: type List(Struct(...))
is not Struct, Map, or Null
```
This is a plan-correctness bug, not a security issue.
## Upstream status
The bug was reported and fixed upstream:
- apache/datafusion#22615 describes `PushDownLeafProjections` crossing
`Unnest`, including the same-named pre/post-unnest column problem.
- apache/datafusion#22620 fixes it by treating `Unnest` as a semantic
barrier for leaf-projection pushdown.
- apache/datafusion#22783 subsequently makes the `LogicalPlan::Unnest`
expression/rebuild contracts consistent while retaining the semantic barrier.
The reduced reproducer fails with DataFusion 54.1.0 and succeeds on
DataFusion `main` at `e1942b144`.
## Temporary SedonaDB workaround
Until SedonaDB upgrades to a DataFusion release containing
apache/datafusion#22620, `sedona-query-planner` replaces DataFusion's optimizer
rule named `push_down_leaf_projections` with a compatibility rule that prevents
leaf projection pushdown across `Unnest`.
Regression coverage includes both a DataFusion-only `List<Struct>` plan and
SedonaDB's `ST_Dump → unnest → geom` workflow.
## Action item
- [ ] After upgrading DataFusion to a release containing
apache/datafusion#22620, remove the vendored `PushDownLeafProjections`
compatibility rule and its registration, while retaining appropriate regression
coverage against the upstream rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
