[
https://issues.apache.org/jira/browse/SPARK-32018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169132#comment-17169132
]
Sunitha Kambhampati commented on SPARK-32018:
---------------------------------------------
[@cloud-fan|https://github.com/cloud-fan], I noticed the back ports now. This
change is more far reaching in its impact as previous callers of
UnsafeRow.getDecimal that would have thrown an exception earlier would now
return null.
As an e.g, a caller like aggregate sum will need changes to account for this.
Earlier cases where sum would throw error for overflow will *now return
incorrect results*. The new tests that were added for sum overflow cases in the
DataFrameSuite in master can be used to see repro.
IMO, it would be better to not back port the setDecimal change in isolation.
wdyt? Please share your thoughts. Thanks.
I added a comment on the pr but since it is closed, adding a comment here.
> Fix UnsafeRow set overflowed decimal
> ------------------------------------
>
> Key: SPARK-32018
> URL: https://issues.apache.org/jira/browse/SPARK-32018
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.6, 3.0.0
> Reporter: Allison Wang
> Assignee: Wenchen Fan
> Priority: Major
> Fix For: 2.4.7, 3.0.1, 3.1.0
>
>
> There is a bug that writing an overflowed decimal into UnsafeRow is fine but
> reading it out will throw ArithmeticException. This exception is thrown when
> calling {{getDecimal}} in UnsafeRow with input decimal's precision greater
> than the input precision. Setting the value of the overflowed decimal to null
> when writing into UnsafeRow should fix this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
