andygrove opened a new pull request, #7251: URL: https://github.com/apache/datafusion-comet/pull/7251
## Which issue does this PR close? Closes #5125. ## Rationale for this change `GenerateExec` runs natively only for `explode` and `posexplode`. `inline` (expanding an array of structs into columns) and `stack` (the standard unpivot) fall back to Spark and take the surrounding stage with them, often directly above a Parquet scan. ## What changes are included in this PR? - **`inline` / `inline_outer`**: a new `inline` flag on the `Explode` proto message. The native planner unnests the array as before, then has `ExplodeExec` flatten the struct into one column per field, using the struct unnesting that the `UnnestExec` fork already carries. Every generated column is reported nullable. - **Null struct elements**: `flatten_struct_cols` now folds a struct's null mask into each field it flattens. Arrow leaves field values under a null struct unspecified, and Spark's `inline` emits a row of nulls for a null element. - **`stack`**: the serde rewrites `stack(n, v1, ..., vk)` into `inline` over `array(named_struct(...), ...)`, one struct per output row, padding the last row with null literals of the column type. Each argument is still evaluated once per input row, as in Spark. If an argument cannot be converted, the operator falls back with the reason `stack arguments are not supported natively`, since the synthesized array is not part of the plan and its own reason would not surface. - Docs: the expression and operator support pages, `understanding-comet-plans.md`, and audit notes for the three generators. `Inline` and `Stack` behave the same in Spark 3.4.3, 3.5.8, 4.0.1 and 4.1.1. ## How are these changes tested? - New SQL file tests `expressions/array/inline.sql` and `expressions/misc/stack.sql`, which assert native execution and match Spark. They cover NULL, empty and NULL-element arrays, NULL fields, the `_outer` variant, `LATERAL VIEW`, a struct field named like a carried column, nested and wide field types, arrays built in the query, uneven `stack` argument counts, NULL literal arguments, and filters and aggregates above the generator. - A Rust unit test for `flatten_struct_cols` with a null struct over non-null field values, including a sliced input. - A `CometGenerateExecSuite` test for the `stack` fallback reason. - Ran locally on the default Spark 4.1 profile: `CometGenerateExecSuite` (54 passed), the `inline`, `stack`, `explode` and `posexplode` SQL files, and the `explode` Rust tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
