[
https://issues.apache.org/jira/browse/SPARK-8800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627306#comment-14627306
]
Jihong MA commented on SPARK-8800:
----------------------------------
I applied the fix and noticed the same.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
stage 1198.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1198.0
(TID 3847, localhost): java.lang.ArithmeticException: Non-terminating decimal
expansion; no exact representable decimal result.^[[0m
^[[31m at java.math.BigDecimal.divide(BigDecimal.java:1616)^[[0m
^[[31m at java.math.BigDecimal.divide(BigDecimal.java:1650)^[[0m
^[[31m at scala.math.BigDecimal.$div(BigDecimal.scala:256)^[[0m
^[[31m at
org.apache.spark.sql.types.Decimal.$div(Decimal.scala:282)^[[0m
^[[31m at
org.apache.spark.sql.types.Decimal$DecimalIsFractional$.div(Decimal.scala:348)^[[0m
^[[31m at
org.apache.spark.sql.types.Decimal$DecimalIsFractional$.div(Decimal.scala:347)^[[0m
^[[31m at
org.apache.spark.sql.catalyst.expressions.Divide$$anonfun$div$1.apply(arithmetic.scala:193)^[[0m
^[[31m at
org.apache.spark.sql.catalyst.expressions.Divide.eval(arithmetic.scala:206)^[[0m
> Spark SQL Decimal Division operation loss of precision/scale when type is
> defined as DecimalType.Unlimited
> ----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-8800
> URL: https://issues.apache.org/jira/browse/SPARK-8800
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Jihong MA
> Assignee: Liang-Chi Hsieh
> Priority: Blocker
> Fix For: 1.5.0
>
>
> According to specification defined in Java doc over BigDecimal :
> http://docs.oracle.com/javase/1.5.0/docs/api/java/math/BigDecimal.html
> When a MathContext object is supplied with a precision setting of 0 (for
> example, MathContext.UNLIMITED), arithmetic operations are exact, as are the
> arithmetic methods which take no MathContext object. (This is the only
> behavior that was supported in releases prior to 5.) As a corollary of
> computing the exact result, the rounding mode setting of a MathContext object
> with a precision setting of 0 is not used and thus irrelevant. In the case of
> divide, the exact quotient could have an infinitely long decimal expansion;
> for example, 1 divided by 3. If the quotient has a nonterminating decimal
> expansion and the operation is specified to return an exact result, an
> ArithmeticException is thrown. Otherwise, the exact result of the division is
> returned, as done for other operations.
> when Decimal data is defined as DecimalType.Unlimited in Spark SQL, the exact
> result of the division should be returned or truncated to precision = 38
> which is in align with what Hive supports. the current behavior is as shown
> following, which cause we lose the accuracy of Decimal division operation.
> scala> val aa = Decimal(2) / Decimal(3);
> aa: org.apache.spark.sql.types.Decimal = 1
> here is another example where we should return 0.125 instead of 0
> scala> val aa = Decimal(1) /Decimal(8)
> aa: org.apache.spark.sql.types.Decimal = 0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
