[
https://issues.apache.org/jira/browse/SPARK-26038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16868926#comment-16868926
]
Josh Rosen edited comment on SPARK-26038 at 6/20/19 8:47 PM:
-------------------------------------------------------------
We just independently rediscovered this bug. I'm adding the {{correctness}}
label because the old behavior gave a wrong answer (as opposed to simply
crashing). Here's an example illustrating the old, buggy behavior:
{code:java}
val largeBigInt = scala.math.BigInt("1234567890123456789" * 2)
val incorrectLongValue = largeBigInt.longValue()
val defaultEncoder: org.apache.spark.sql.Encoder[scala.math.BigInt] =
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder[BigInt]
val ds =
spark.createDataset(Seq(largeBigInt))(defaultEncoder).map(identity(_))(defaultEncoder)
val roundtrippedThroughEncoder = ds.first()
println(roundtrippedThroughEncoder == largeBigInt){code}
In Spark 2.4.0, this outputs:
{code:java}
roundtrippedThroughEncoder: scala.math.BigInt = -3191638190864629483{code}
and the equality comparison is {{false}}.
[~smilegator] and [~juliuszsompolski], how do you feel about backporting this
for 2.4.4?
was (Author: joshrosen):
We just independently rediscovered this bug. I'm adding the {{correctness}}
label because the old behavior gave wrong answer (as opposed to simply
crashing). Here's an example illustrating the old, buggy behavior:
{code:java}
val largeBigInt = scala.math.BigInt("1234567890123456789" * 2)
val incorrectLongValue = largeBigInt.longValue()
val defaultEncoder: org.apache.spark.sql.Encoder[scala.math.BigInt] =
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder[BigInt]
val ds =
spark.createDataset(Seq(largeBigInt))(defaultEncoder).map(identity(_))(defaultEncoder)
val roundtrippedThroughEncoder = ds.first()
println(roundtrippedThroughEncoder == largeBigInt){code}
In Spark 2.4.0, this outputs:
{code:java}
roundtrippedThroughEncoder: scala.math.BigInt = -3191638190864629483{code}
and the equality comparison is {{false}}.
[~smilegator] and [~juliuszsompolski], how do you feel about backporting this
for 2.4.4?
> Decimal toScalaBigInt/toJavaBigInteger not work for decimals not fitting in
> long
> --------------------------------------------------------------------------------
>
> Key: SPARK-26038
> URL: https://issues.apache.org/jira/browse/SPARK-26038
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0, 2.3.0, 2.4.0
> Reporter: Juliusz Sompolski
> Assignee: Juliusz Sompolski
> Priority: Major
> Labels: correctness
> Fix For: 3.0.0
>
>
> Decimal toScalaBigInt/toJavaBigInteger just called toLong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
