[
https://issues.apache.org/jira/browse/SPARK-59505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115346#comment-18115346
]
Li Guo commented on SPARK-59505:
--------------------------------
Opened a PR for this: https://github.com/apache/spark/pull/58797
[~podongfeng] this sits in the decimal rescaling from SPARK-53938, so I would
appreciate your eyes on it when you get a chance. The change is one line
(ROUND_HALF_EVEN to ROUND_HALF_UP); the rest is tests covering the Arrow UDF,
the Arrow UDTF, a Python data source and createDataFrame in Classic and
Connect. cc [~gurwls223]
> Arrow conversion rounds decimals HALF_EVEN, so Arrow Python UDFs and Connect
> createDataFrame return different values than SQL and Classic
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59505
> URL: https://issues.apache.org/jira/browse/SPARK-59505
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 4.2.0, 4.3.0, 5.0.0, 4.1.3
> Reporter: Li Guo
> Priority: Major
> Labels: pull-request-available
>
> {{LocalDataToArrowConversion}} rescales a Python {{Decimal}} to the declared
> scale with {{decimal.Context(prec=..., rounding=decimal.ROUND_HALF_EVEN)}}
> (python/pyspark/sql/conversion.py, added by SPARK-53938). The JVM rounds
> HALF_UP whenever it rescales a decimal ({{Decimal.set}},
> {{Decimal.changePrecision}}, {{CAST(x AS DECIMAL(p, s))}}), and the pickled
> Python UDF and UDTF paths and Classic {{createDataFrame}} inherit that
> because the JVM does the rescale for them. So the Arrow-optimized Python UDF
> (the default since 4.2), Arrow UDTFs, Python data sources and Spark Connect's
> {{createDataFrame}} produce a different number than SQL for any value that
> sits on a rounding tie.
> {code:python}
> from decimal import Decimal
> from pyspark.sql.functions import col, udf
> from pyspark.sql.types import DecimalType
> df = spark.sql("SELECT * FROM VALUES ('1.005'), ('1.025'), ('0.125') AS t(v)")
> [r[0] for r in df.select(col("v").cast(DecimalType(20, 2))).collect()]
> # [Decimal('1.01'), Decimal('1.03'), Decimal('0.13')]
> f = udf(lambda v: Decimal(v), DecimalType(20, 2), useArrow=True) # the
> default since 4.2
> [r[0] for r in df.select(f("v")).collect()]
> # [Decimal('1.00'), Decimal('1.02'), Decimal('0.12')]
> f2 = udf(lambda v: Decimal(v), DecimalType(20, 2), useArrow=False)
> [r[0] for r in df.select(f2("v")).collect()]
> # [Decimal('1.01'), Decimal('1.03'), Decimal('0.13')]
> {code}
> Spark Connect and Classic also disagree on the same input:
> {{spark.createDataFrame([(Decimal("1.005"),)], "d decimal(20, 2)")}} gives
> 1.01 in Classic and 1.00 through a Connect session. Reproduced on released
> 4.2.0 and on master, in Classic and Connect sessions.
> Fix: use {{ROUND_HALF_UP}} in that context. One line plus tests on the
> converter, the Arrow UDF, the Arrow UDTF, a Python data source and
> createDataFrame in both modes. branch-4.1, branch-4.2 and branch-4.3 carry
> the same line, so the fix backports cleanly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
