andygrove opened a new issue, #7023:
URL: https://github.com/apache/datafusion-comet/issues/7023
Part of #5572. Follow-up from #5050.
With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, a negative-scale
decimal input falls the whole projection back to Spark for four serdes:
- `CometCeil` and `CometFloor` return `Unsupported("Decimal type $t has
negative scale")`
([math.scala#L44-L45](https://github.com/apache/datafusion-comet/blob/66c63bd307b1b11fb684e6b89e9bb220f444de7c/spark/src/main/scala/org/apache/comet/serde/math.scala#L44-L45),
[#L68-L69](https://github.com/apache/datafusion-comet/blob/66c63bd307b1b11fb684e6b89e9bb220f444de7c/spark/src/main/scala/org/apache/comet/serde/math.scala#L68-L69)).
- Once #5050 lands, `CometDivide` and `CometIntegralDivide` return
`Unsupported(negScaleDecimalArithmeticReason)` from
`MathBase.negScaleDecimalRejection`
([arithmetic.scala#L105-L119](https://github.com/apache/datafusion-comet/blob/29c4509a5b609ddf7d0706a69d213f6339bad161/spark/src/main/scala/org/apache/comet/serde/arithmetic.scala#L105-L118)).
Their native kernel can't take a negative scale: `decimal_div` computes its
scale-alignment exponents in `u32`, and #5050 makes it reject one.
`CometRound` returns `Unsupported` for the same input but mixes in
`CodegenDispatchFallback`, so `round` stays in the Comet pipeline and runs
Spark's own `doGenCode`
([arithmetic.scala#L366-L382](https://github.com/apache/datafusion-comet/blob/66c63bd307b1b11fb684e6b89e9bb220f444de7c/spark/src/main/scala/org/apache/comet/serde/arithmetic.scala#L366-L382)).
`round on negative-scale decimal` in `CometExpressionSuite` covers it. The
other four could do the same. The EPIC deferred `ceil` and `floor` as marginal,
since only the legacy flag reaches them, and #5050 adds division to that group.
Fix: for `CometCeil` and `CometFloor` the negative scale is the only
`Unsupported` case, so mixing in `CodegenDispatchFallback` is the whole change.
For `CometDivide` and `CometIntegralDivide` it isn't a one-line change. The
mixin would also send their other `Unsupported` results to the dispatcher,
namely `mathDataTypeSupportLevel`'s unsupported-datatype case and
`CometDivide`'s check that a decimal result has decimal operands. Each of those
needs checking against the dispatcher's type gate, or these serdes need a way
to dispatch only the negative-scale case.
Tests should cover a materialized negative-scale column, not only a cast
literal. That input broke the dispatcher's decimal reader until #5050 made it
read negative scales through `BigDecimal`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
