[
https://issues.apache.org/jira/browse/SPARK-29123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933560#comment-16933560
]
Marco Gaido commented on SPARK-29123:
-------------------------------------
[~benny] the point here is: Spark can represent decimals with max precision 38.
When defining the result type multiple options are possible.
We decided to follow SQLServer implementation by default (you can check their
docs too). There isn't much more docs as tehre isn't much more to say. Since
the precision is limited, there are 2 options:
- in case you do not allow precision loss, an overflow can happen. In that
case Spark returns NULL.
- in case you allow precision loss, precision loss is preferred over overflow.
This is the behavior SQLServer has and it is ANSI compliant.
You can see the PR for SPARK-22036 for more details.
> DecimalType multiplication precision loss
> ------------------------------------------
>
> Key: SPARK-29123
> URL: https://issues.apache.org/jira/browse/SPARK-29123
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.4.3
> Reporter: Benny Lu
> Priority: Major
>
> When doing multiplication with PySpark, it seems PySpark is losing precision.
> For example, when multiplying two decimals with precision 38,10, it returns
> 38,6 instead of 38,10. It also truncates result to three decimals which is
> incorrect result.
> {code:java}
> from decimal import Decimal
> from pyspark.sql.types import DecimalType, StructType, StructField
> schema = StructType([StructField("amount", DecimalType(38,10)),
> StructField("fx", DecimalType(38,10))])
> df = spark.createDataFrame([(Decimal(233.00), Decimal(1.1403218880))],
> schema=schema)
> df.printSchema()
> df = df.withColumn("amount_usd", df.amount * df.fx)
> df.printSchema()
> df.show()
> {code}
> Result
> {code:java}
> >>> df.printSchema()
> root
> |-- amount: decimal(38,10) (nullable = true)
> |-- fx: decimal(38,10) (nullable = true)
> |-- amount_usd: decimal(38,6) (nullable = true)
> >>> df = df.withColumn("amount_usd", df.amount * df.fx)
> >>> df.printSchema()
> root
> |-- amount: decimal(38,10) (nullable = true)
> |-- fx: decimal(38,10) (nullable = true)
> |-- amount_usd: decimal(38,6) (nullable = true)
> >>> df.show()
> +--------------+------------+----------+
> | amount| fx|amount_usd|
> +--------------+------------+----------+
> |233.0000000000|1.1403218880|265.695000|
> +--------------+------------+----------+
> {code}
> When rounding to two decimals, it returns 265.70 but the correct result
> should be 265.69499 and when rounded to two decimals, it should be 265.69.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
