mgaido91 commented on a change in pull request #25136: [SPARK-28322][SQL] Add
support to Decimal type for integral divide
URL: https://github.com/apache/spark/pull/25136#discussion_r312702907
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala
##########
@@ -174,6 +174,20 @@ object DecimalPrecision extends TypeCoercionRule {
CheckOverflow(Pmod(promotePrecision(e1, widerType), promotePrecision(e2,
widerType)),
resultType, nullOnOverflow)
+ case expr @ IntegralDivide(
+ e1 @ DecimalType.Expression(p1, s1), e2 @ DecimalType.Expression(p2,
s2)) =>
+ val widerType = widerDecimalType(p1, s1, p2, s2)
+ val promotedExpr =
+ IntegralDivide(promotePrecision(e1, widerType), promotePrecision(e2,
widerType))
+ if (expr.dataType.isInstanceOf[DecimalType]) {
+ // This follows division rule
+ val intDig = p1 - s1 + s2
+ // No precision loss can happen as the result scale is 0, so only
overflow can happen
+ CheckOverflow(promotedExpr, DecimalType.bounded(intDig, 0),
nullOnOverflow)
Review comment:
when `spark.sql.legacy.integralDivide.returnBigint=true` anyway, the
overflow should be checked by the `toLong` method, ie. the "cast" to long,
which is yet to be handled in a different PR.
The `CheckOverflow` is still needed to return the proper datatype here. Eg,
if the operands are `decimal(35, 0)` and `decimal(5, 3)`, they are promoted to
`decimal(35, 3)`, but the correct return type should be `decimal(38,0)` which
prevents any overflow. So despite no exception or null can happen, the
`CheckOverflow` is still needed. I added a comment for this. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
