Li Guo created SPARK-59505:
------------------------------
Summary: Arrow conversion rounds decimals HALF_EVEN, so Arrow
Python UDFs and Connect createDataFrame return different values than SQL and
Classic
Key: SPARK-59505
URL: https://issues.apache.org/jira/browse/SPARK-59505
Project: Spark
Issue Type: Bug
Components: Connect, PySpark
Affects Versions: 4.1.3, 4.2.0, 4.3.0, 5.0.0
Reporter: Li Guo
{{LocalDataToArrowConversion}} rescales a Python {{Decimal}} to the declared
scale with {{decimal.Context(prec=..., rounding=decimal.ROUND_HALF_EVEN)}}
(python/pyspark/sql/conversion.py, added by SPARK-53938). The JVM rounds
HALF_UP whenever it rescales a decimal ({{Decimal.set}},
{{Decimal.changePrecision}}, {{CAST(x AS DECIMAL(p, s))}}), and the pickled
Python UDF and UDTF paths and Classic {{createDataFrame}} inherit that because
the JVM does the rescale for them. So the Arrow-optimized Python UDF (the
default since 4.2), Arrow UDTFs, Python data sources and Spark Connect's
{{createDataFrame}} produce a different number than SQL for any value that sits
on a rounding tie.
{code:python}
from decimal import Decimal
from pyspark.sql.functions import col, udf
from pyspark.sql.types import DecimalType
df = spark.sql("SELECT * FROM VALUES ('1.005'), ('1.025'), ('0.125') AS t(v)")
[r[0] for r in df.select(col("v").cast(DecimalType(20, 2))).collect()]
# [Decimal('1.01'), Decimal('1.03'), Decimal('0.13')]
f = udf(lambda v: Decimal(v), DecimalType(20, 2), useArrow=True) # the
default since 4.2
[r[0] for r in df.select(f("v")).collect()]
# [Decimal('1.00'), Decimal('1.02'), Decimal('0.12')]
f2 = udf(lambda v: Decimal(v), DecimalType(20, 2), useArrow=False)
[r[0] for r in df.select(f2("v")).collect()]
# [Decimal('1.01'), Decimal('1.03'), Decimal('0.13')]
{code}
Spark Connect and Classic also disagree on the same input:
{{spark.createDataFrame([(Decimal("1.005"),)], "d decimal(20, 2)")}} gives 1.01
in Classic and 1.00 through a Connect session. Reproduced on released 4.2.0 and
on master, in Classic and Connect sessions.
Fix: use {{ROUND_HALF_UP}} in that context. One line plus tests on the
converter, the Arrow UDF, the Arrow UDTF, a Python data source and
createDataFrame in both modes. branch-4.1, branch-4.2 and branch-4.3 carry the
same line, so the fix backports cleanly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
