andygrove opened a new issue, #7063: URL: https://github.com/apache/datafusion-comet/issues/7063
### Describe the bug The native Murmur3 and xxHash64 hashers have no branch for Arrow `Null`, `Interval(YearMonth)` or `Duration(Microsecond)`, the types Comet maps Spark's `NullType`, `YearMonthIntervalType` and `DayTimeIntervalType` to. A value of any of them falls through to the catch-all in `create_hashes_internal!` and fails the query: ``` org.apache.comet.CometNativeException: Unsupported data type in hasher: Null. This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues at org.apache.comet.Native.executePlan(Native Method) at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:260) ``` https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/native/spark-expr/src/hash_funcs/utils.rs#L1272-L1278 Two Scala gates let these types reach it. `hash.scala` checks a hash expression's children with `QueryPlanSerde.supportedDataType`, which accepts `NullType` at any depth. So with the default configuration, `hash(a, null)`, `xxhash64(a, null)` and `hash(named_struct('x', a, 'n', null))` run natively and fail. For `hash(a, null)` Spark returns the same value as `hash(a)`. The ANSI interval types are rejected there, so hashing them falls back to Spark. https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/spark/src/main/scala/org/apache/comet/serde/hash.scala#L136-L147 Native round-robin shuffle places rows by hashing every data column with the same kernel, and the native shuffle's type check accepts `NullType` and both ANSI interval types as data columns. So with `spark.comet.shuffle.native.partitioning.roundrobin.enabled=true`, a `repartition(n)` over a native plan with a column of any of the three types, at any depth, fails with `Error inserting batch: Internal error: Unsupported data type in hasher: ...`. With the config off, which is the default, those shuffles run on JVM columnar shuffle or Spark shuffle and work. https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala#L662-L673 https://github.com/apache/datafusion-comet/blob/3a4a15940983e4bb9a9a74c816ee51e2cf6d9e17/native/shuffle/src/partitioners/multi_partition.rs#L419-L434 ### Steps to reproduce ```sql CREATE TABLE t (a INT) USING parquet; INSERT INTO t SELECT CAST(id AS INT) FROM range(1000); -- default configuration SELECT a, hash(a, null) AS h FROM t; SELECT a, xxhash64(a, null) AS h FROM t; SELECT a, hash(named_struct('x', a, 'n', null)) AS h FROM t; -- native round robin SET spark.comet.shuffle.native.partitioning.roundrobin.enabled=true; SELECT /*+ REPARTITION(4) */ a, null AS n FROM t; SELECT /*+ REPARTITION(4) */ a, make_ym_interval(a % 5, 1) AS ym FROM t; SELECT /*+ REPARTITION(4) */ a, make_dt_interval(a % 5, 1, 2, 3.5) AS dt FROM t; ``` Every query fails. I reproduced them on c3316fbe39 with the Spark 4.1 profile, and the code involved is unchanged on main at 3a4a159409. ### Expected behavior The queries should return the same results as Spark. Spark's `HashExpression` leaves the hash unchanged for `NullType`, hashes `YearMonthIntervalType` as an int and hashes `DayTimeIntervalType` as a long. The native side can match that exactly with a no-op arm for `Null` and the existing primitive int and long paths for `Interval(YearMonth)` and `Duration(Microsecond)`, which fixes both entry points. With those arms in place, `hash.scala` could also accept the ANSI interval types, which #5061 lists as falling back today. ### Additional context #5059 was the same failure for `Interval(MonthDayNano)`, fixed in #5135. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
