[
https://issues.apache.org/jira/browse/SPARK-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14610831#comment-14610831
]
Jihong MA commented on SPARK-8677:
----------------------------------
Thanks for fixing the division problem. but this fix introduces one more issue
w.r.t the accuracy of Decimal computation.
scala> val aa = Decimal(2) / Decimal(3);
aa: org.apache.spark.sql.types.Decimal = 1
when a Decimal is defined as Decimal.Unlimited, we are not expecting the
division result's scale value to inherit from its parent, this is causing big
accuracy issue once we go coupe round of division over decimal data vs. double
data. below is a sample output from my run.
10:27:46.042 WARN
org.apache.spark.sql.catalyst.expressions.CombinePartialStdFunction: COMBINE
STDDEV DOUBLE-------4.0 , 0.8VALUE
10:27:46.137 WARN
org.apache.spark.sql.catalyst.expressions.CombinePartialStdFunction: COMBINE
STDDEV DECIMAL-------4.29000 , 0.858VALUE
> Decimal divide operation throws ArithmeticException
> ---------------------------------------------------
>
> Key: SPARK-8677
> URL: https://issues.apache.org/jira/browse/SPARK-8677
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Liang-Chi Hsieh
> Assignee: Liang-Chi Hsieh
> Fix For: 1.5.0
>
>
> Please refer to [BigDecimal
> doc|http://docs.oracle.com/javase/1.5.0/docs/api/java/math/BigDecimal.html]:
> {quote}
> ... the rounding mode setting of a MathContext object with a precision
> setting of 0 is not used and thus irrelevant. In the case of divide, the
> exact quotient could have an infinitely long decimal expansion; for example,
> 1 divided by 3.
> {quote}
> Because we provide a MathContext.UNLIMITED in toBigDecimal, Decimal divide
> operation will throw the following exception:
> {code}
> val decimal = Decimal(1.0, 10, 3) / Decimal(3.0, 10, 3)
> [info] java.lang.ArithmeticException: Non-terminating decimal expansion; no
> exact representable decimal result.
> [info] at java.math.BigDecimal.divide(BigDecimal.java:1690)
> [info] at java.math.BigDecimal.divide(BigDecimal.java:1723)
> [info] at scala.math.BigDecimal.$div(BigDecimal.scala:256)
> [info] at org.apache.spark.sql.types.Decimal.$div(Decimal.scala:272)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
