comphead opened a new pull request, #26266: URL: https://github.com/apache/datafusion/pull/26266
## Which issue does this PR close? - Part of #15914 - Related Comet issue: https://github.com/apache/datafusion-comet/issues/5125 ## Rationale for this change Spark's `stack(n, expr1, ..., exprk)` separates the values into `n` rows. It is the standard way to unpivot in Spark SQL, and Comet falls back to Spark for it. ## What changes are included in this PR? - `stack` in `datafusion-spark`. A ScalarUDF cannot emit rows, so like `json_tuple` it returns the `n` rows as a `List<Struct<col0, col1, ...>>`, which is the data type of Spark's Generator. The caller (Comet, or `unnest` in SQL) expands it. - The arguments are validated as in Spark's `Stack.checkInputDataTypes`, with the same messages. Only untyped NULLs are coerced, to the type of their column. - Primitive columns are copied with a strided loop, and columns of only scalars (such as unpivot keys) are repeated with `concat` doubling. Other types use `interleave`. On the new `stack` bench this is 3x to 5x faster than `interleave` alone. ## What is the testing strategy for this PR? `spark/generator/stack.slt` ports the cases of Spark's `GeneratorFunctionSuite`, `GeneratorExpressionSuite` and `TypeCoercionSuite`, and of the `generators`, `table-valued-functions` and `join-lateral` golden files. Unit tests check the kernels against `interleave` on empty and multi-word inputs, and the list size overflow. ## Are there any user-facing changes? A new Spark-compatible `stack` function. Unlike Spark, `n` must be an integer literal, so a foldable expression such as `3 - 1` is rejected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
