linhr commented on issue #19692: URL: https://github.com/apache/datafusion/issues/19692#issuecomment-3976767621
We have started a preliminary upgrade in Sail. This time, we are trying a purely AI-driven approach. The progress so far can be found here: <https://github.com/lakehq/sail/pull/1442>. We are targeting DataFusion commit `5d8249f` for now. The code was able to compile, but there were many test failures, which mostly came with error like `Execution error: Cannot access field at argument 1: type List(non-null Struct(XXX)) is not Struct, Map, or Null`. This is what Claude suggested: > Root cause: The new DataFusion optimizer rule `push_down_leaf_projections` (new in rev `5d8249f`) moves `get_field` expressions closer to data sources. It incorrectly pushes them through `Unnest` nodes — where the column `_field_N` still has type `List(Struct(...))` (pre-unnest), not `Struct(...)` (post-unnest) — causing the error: `Cannot access field at argument 1: type List(Struct(...)) is not Struct, Map, or Null`. Note that we haven't validated the correctness of this AI comment, nor are we sure that this captures the root cause. I'm just sharing it here in case anyone familiar with this topic has some pointers to consider. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
