[
https://issues.apache.org/jira/browse/SPARK-8342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14584784#comment-14584784
]
Rene Treffer commented on SPARK-8342:
-------------------------------------
Oh no, I managed to create a completely unrelated testcase for my problem.
Reduction ftw....
Here is what breaks in the >2^112 range
{code}
import org.apache.spark.sql.types.Decimal
val one = Decimal(1)
val two = Decimal(2)
def pow(n : Int) : Decimal = if (n <= 0) { one } else {
val a = pow(n - 1)
a.changePrecision(n,0)
two.changePrecision(n,0)
a * two
}
(109 to 120).foreach(n =>
println(pow(n).toJavaBigDecimal.unscaledValue.toString))
649037107316853453566312041152512
1298074214633706907132624082305024
2596148429267413814265248164610048
5192296858534827628530496329220096
1038459371706965525706099265844019
2076918743413931051412198531688038
4153837486827862102824397063376076
8307674973655724205648794126752152
1661534994731144841129758825350430
3323069989462289682259517650700860
6646139978924579364519035301401720
1329227995784915872903807060280344
{code}
So precision gets lost. It's not visible on the normal output path, but it is a
loss of precision (10^n is always enough to hold 2^n without loss of precision).
Should I open another ticket for this?
> Decimal Math beyond ~2^112 is broken
> ------------------------------------
>
> Key: SPARK-8342
> URL: https://issues.apache.org/jira/browse/SPARK-8342
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Rene Treffer
>
> Here is a snippet from the spark-shell that should not happen
> {code}
> scala> val d = Decimal(Long.MaxValue,100,0) * Decimal(Long.MaxValue,100,0)
> d: org.apache.spark.sql.types.Decimal = 0
> scala> d.toDebugString
> res3: String = Decimal(expanded,0,1,0})
> {code}
> It looks like precision gets reseted on some operations and values are then
> truncated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
