peterxcli opened a new issue, #5675:
URL: https://github.com/apache/datafusion-comet/issues/5675
### Describe the bug
Spark's `ConcatWs` accepts both `StringType` and `ArrayType(StringType)`
children and flattens array arguments: `concat_ws(',', array('a','b'), 'c')` =
`a,b,c`. `CometConcatWs.getSupportLevel`
(`spark/src/main/scala/org/apache/comet/serde/strings.scala:291-298`) never
inspects child types and lowers every non-foldable call to DataFusion's
`concat_ws`, which accepts only `Utf8` / `LargeUtf8` / `Utf8View` and fails
with `Input was List(...) which is not a supported datatype for concat_ws
function` (datafusion-functions 54.1 `concat_ws.rs`). The query fails at native
execution where Spark returns a result. There is no Comet test with an array
argument.
### Steps to reproduce
```sql
CREATE TABLE t USING parquet AS SELECT * FROM VALUES (array('a','b'), 'c')
AS t(arr, s);
SELECT concat_ws(',', arr, s) FROM t;
-- also: SELECT concat_ws(',', split(s, ' ')) FROM t
```
Spark: `a,b,c`. Comet: native error `Input was List(Field { name: "item",
data_type: Utf8, ... }) which is not a supported datatype for concat_ws
function`.
### Expected behavior
`a,b,c` — either flatten array arguments natively like Spark, or fall back
to Spark for them.
### Proposed solution
Short term: in `getSupportLevel`, return `Unsupported` (with a reason) when
any child's `dataType` is an `ArrayType`, so the expression falls back (or goes
through the codegen dispatcher). Longer term: a Spark-compatible native
`concat_ws` that flattens list arguments (skipping null elements like Spark).
Add a test with an array argument.
### Additional context
Only the all-foldable case is declined today (and Spark constant-folds that
anyway). Related: #3339 (null-separator crash, fixed by #3542 with a
null-literal branch but no argument-type gate).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
