andygrove opened a new issue, #5743:
URL: https://github.com/apache/datafusion-comet/issues/5743
### Describe the bug
`slice` crashes natively when its input array has a **non-nullable element
field**. `spark_array_slice` passes the input's child field through to its
output, but the return type Comet promises at planning time has a nullable
element, so the two disagree and execution fails.
Spark declares several expressions with `containsNull = false`, and any of
them feeding `slice` reproduces this. `StringSplit.dataType` is
`ArrayType(StringType, containsNull = false)` and `Sequence.dataType` is
`ArrayType(start.dataType, containsNull = false)`, so both are enough on their
own.
This fails on **release** builds too, not just debug. The two builds surface
it differently:
- debug: `Assertion failed: result_data_type == *expected_type: Function
'spark_array_slice' returned value of type 'List(non-null Utf8)' while the
following type was promised at planning time and expected: 'List(Utf8)'.`
- release: `Invalid argument error: column types must match schema types,
expected List(Utf8) but found List(non-null Utf8) at column index 1`
This is the same assertion as #4789, but by a route that issue explicitly
ruled out. Its body notes the crash "does NOT reproduce over a native Parquet
scan (which normalizes children to nullable)". The repro below is over a native
Parquet scan, and the non-nullable child comes from the expression rather than
from the scan, so the #4789 fix does not cover it.
### Steps to reproduce
In a suite extending `CometTestBase`:
```scala
test("slice over a non-nullable element array") {
val input = Seq((1, "axb", 2), (2, "", 3), (3, "cxd", 2))
withParquetDataFrame(input) { parquet =>
withParquetTable(parquet.toDF("id", "s", "n"), "t") {
// both of these fail
checkSparkAnswerAndOperator(sql("SELECT id, slice(split(s, 'x'), 1, n)
AS a FROM t"))
checkSparkAnswerAndOperator(sql("SELECT id, slice(sequence(1, n), 1,
2) AS a FROM t"))
}
}
}
```
No generator, no local table scan, and no shuffle is involved. A bare
`CometProject` over `CometNativeScan` is enough.
### Scope
`slice` is the only affected kernel I found. Other native array kernels
handle a non-nullable element field correctly, so this looks specific to how
`CometSlice` derives its return type rather than a general problem with
non-nullable children:
| expression | result |
| --- | --- |
| `slice(split(s, 'x'), 1, n)` | **FAIL** |
| `slice(sequence(1, n), 1, 2)` | **FAIL** |
| `slice(concat(split(s, 'x'), array('z')), 1, n)` | **FAIL** |
| `slice(array(1, 2, 3), 1, n)` (nullable elements) | OK |
| `slice(arr, 1, n)` over a stored `array<string>` | OK |
| `array_remove(sequence(1, n), 2)` | OK |
| `reverse(sequence(1, n))` | OK |
| `array_append(sequence(1, n), 9)` | OK |
| `concat(sequence(1, n), array(9))` | OK |
| `sort_array(sequence(1, n))` | OK |
| `size(sequence(1, n))` | OK |
Setting `spark.comet.expression.StringSplit.allowIncompatible=true` does
**not** help, so this is not specific to the JVM codegen dispatcher. The native
`split` produces the same non-nullable element field, which is expected given
Spark's declared type.
### Expected behavior
Same results as Spark, no native crash. Either `spark_array_slice` should
normalize its output child field to match the return type Comet planned, or
`CometSlice` should declare a return type whose element nullability matches the
input it will actually receive.
### Additional context
Found while reviewing #5737, which is unrelated. That PR only changes the
`Explode` arm of the planner, and the crash reproduces with no generator in the
plan at all.
Reproduced on Spark 4.1.3 / JDK 17, macOS aarch64, at `7190df631`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
