[
https://issues.apache.org/jira/browse/SPARK-25454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16620192#comment-16620192
]
Marco Gaido commented on SPARK-25454:
-------------------------------------
[~bersprockets] you're right, the only "wrong" thing of your statements is that
the problem is not about 1000 but about 1e6, which in 2.2 was considered a
decimal(10, 0) and now it is parsed as a decimal(1, -6). You could reproduce
the same issue using {{lit(BigDecimal(1e6))}} in 2.2. So the problem is that we
are not handling properly decimals with negative scale, but we are not
forbidding their existence either, hence the issue. Making more common the
presence of negative scale numbers made the issue more evident. Hope this is
clear. Thanks.
> Division between operands with negative scale can cause precision loss
> ----------------------------------------------------------------------
>
> Key: SPARK-25454
> URL: https://issues.apache.org/jira/browse/SPARK-25454
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0, 2.3.1
> Reporter: Marco Gaido
> Priority: Major
>
> The issue was originally reported by [~bersprockets] here:
> https://issues.apache.org/jira/browse/SPARK-22036?focusedCommentId=16618104&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16618104.
> The problem consist in a precision loss when the second operand of the
> division is a decimal with a negative scale. It was present also before 2.3
> but it was harder to reproduce: you had to do something like
> {{lit(BigDecimal(100e6))}}, while now this can happen more frequently with
> SQL constants.
> The problem is that our logic is taken from Hive and SQLServer where decimals
> with negative scales are not allowed. We might also consider enforcing this
> too in 3.0 eventually. Meanwhile we can fix the logic for computing the
> result type for a division.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
