andygrove opened a new issue, #5581:
URL: https://github.com/apache/datafusion-comet/issues/5581
Part of #5572.
`HashUtils.unsupportedReasonFor` declines a `DecimalType` with precision >
18, walking into structs, arrays and map key/value types to find one:
```scala
case d: DecimalType if d.precision > 18 => Some(unsupportedDecimalReason)
```
`spark/src/main/scala/org/apache/comet/serde/hash.scala:137`
Four serdes route through it and none mixes in `CodegenDispatchFallback`:
`CometMurmur3Hash` (`hash.scala:49`), `CometXxHash64` (`:27`), `CometSha1`
(`:101`) and `CometSha2` (`:75`). So `hash(high_precision_decimal_col)` or
`xxhash64(...)` fails the whole projection back to Spark, which matters because
these turn up in bucketing, partitioning and dedup paths where the surrounding
operator is worth keeping native.
The reason given — Spark hashes via Java `BigDecimal` — is again a case
where the dispatcher is the right answer rather than a native fix, since
running Spark's `HashExpression.doGenCode` reproduces the `BigDecimal` hashing
exactly. `DecimalType` is in `CometBatchKernelCodegen.isSupportedDataType` at
any precision.
`CometSha2` has a second decline worth covering in the same change:
```scala
if (!expr.right.foldable) {
Unsupported(Some(nonFoldableNumBitsReason))
}
```
`hash.scala:84-85` — a `numBits` argument that is a column rather than a
literal. `Sha2.doGenCode` handles that fine.
Not in scope: the `TimeType` arm of the same helper. `isTimeType` values are
admitted by the kernel via `case dt if isTimeType(dt) => true`, so it may be
dispatchable too, but Comet's `TimeType` support is new enough that it deserves
its own look rather than being folded in here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
