davidlghellin opened a new pull request, #20928:
URL: https://github.com/apache/datafusion/pull/20928
## Which issue does this PR close?
Part of #15914
## Rationale for this change
DataFusion core's `concat_ws` does not support array arguments. Spark's
`concat_ws(sep, ...)` accepts both scalar strings and arrays, expanding array
elements and skipping nulls. This is needed for Spark compatibility in the
`datafusion-spark` crate.
## What changes are included in this PR?
- New `SparkConcatWs` UDF in
`datafusion/spark/src/function/string/concat_ws.rs`
- Supports `concat_ws(sep, str1, str2, ...)` with scalar strings
- Supports array arguments: `concat_ws(',', array('a', 'b'), 'c')` →
`"a,b,c"`
- Null scalars and null array elements are skipped (Spark behavior)
- Null separator returns NULL
- Zero value arguments (`concat_ws(',')`) returns empty string
- Supports Utf8, LargeUtf8, Utf8View, List, and LargeList types
- Registered the function in `mod.rs` (`make_udf_function!`,
`export_functions!`, `functions()`)
- Replaced commented-out SLT tests with 14 working test cases covering basic
usage, arrays, mixed arguments, nulls, column expressions, and edge cases
## Are these changes tested?
Yes.
- 7 unit tests in `concat_ws.rs` (basic, null values skipped, null
separator, list arrays, list with nulls, mixed scalar+list, multiple rows)
- 14 SLT tests in `spark/string/concat_ws.slt` covering scalars, arrays,
nulls, column expressions, and edge cases
## Are there any user-facing changes?
No. This is a new function in the `datafusion-spark` crate only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
