viirya commented on code in PR #59335:
URL: https://github.com/apache/spark/pull/59335#discussion_r4277342207
##########
sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala:
##########
@@ -458,6 +459,20 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
if (dv.ne(null)) {
// We get here if either we started with a BigDecimal, or we switched to
one because we would
// have overflowed our Long; in either case we must rescale dv to the
new scale.
+ if (math.abs(dv.scale.toLong - scale) > DecimalType.MAX_PRECISION) {
+ // setScale's cost grows with the scale change, and it throws once
10^change exceeds
+ // BigInteger's range (e.g. for 1e-2147483647) or the change exceeds
the Int range.
+ // Skip it when the result is known.
+ val numIntegralDigits = dv.precision.toLong - dv.scale
Review Comment:
Following up: I tried switching this to
`numDigitsInIntegralPart(dv.underlying)` after #59334 merged, but calling the
companion's private method from the class makes Scala emit it as public, which
changes the computed `serialVersionUID` of `Decimal$`. `UdfSerializationSuite`
(SPARK-59708) then fails, because that SUID is pinned in
`UdfSerialization.auditedTransitions`. So I reverted to the inline
`dv.precision.toLong - dv.scale`, which keeps `Decimal$` unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
