NestDream opened a new pull request, #58797:
URL: https://github.com/apache/spark/pull/58797
### What changes were proposed in this pull request?
`LocalDataToArrowConversion` rescales a Python `Decimal` to the declared
scale with a `decimal.Context` whose rounding mode is `ROUND_HALF_EVEN`,
Python's default. This PR changes it to `ROUND_HALF_UP`, which is what the JVM
uses whenever it rescales a decimal (`Decimal.set`, `Decimal.changePrecision`,
`CAST` in ANSI and non-ANSI mode), and therefore what the pickled Python UDF
and UDTF paths and Classic `createDataFrame` produce, since the JVM does the
rescale for them. Tests are added for the converter, the Arrow-optimized Python
UDF, the Arrow Python UDTF, a Python data source and `createDataFrame`, with
values that sit exactly on a rounding tie, next to the `test_decimal_round`
tests that SPARK-53938 added. The migration guide gets a note.
The pandas-based paths
(`spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled`, scalar
`pandas_udf`) do not use this converter; they hand the values to pyarrow casts,
which raise on such values with the default
`spark.sql.execution.pandas.convertToArrowArraySafely` and truncate with it
off. They are unchanged here.
### Why are the changes needed?
With the default Arrow-optimized Python UDF, `udf(lambda v: Decimal(v),
DecimalType(20, 2))` returns 1.00 for `'1.005'`, 1.02 for `'1.025'` and 0.12
for `'0.125'`, while `CAST(v AS DECIMAL(20, 2))` and the same UDF with
`useArrow=False` return 1.01, 1.03 and 0.13. Spark Connect's `createDataFrame`
goes through the same converter, so `createDataFrame([(Decimal("1.005"),)], "d
decimal(20, 2)")` gives 1.00 in a Connect session and 1.01 in Classic. Arrow
UDTFs and Python data sources use the converter as well. The rounding was
introduced by SPARK-53938 (#52637), whose goal was to match Classic; the
explicit `Context` arrived there to fix an `InvalidOperation` on 38-digit
values (https://github.com/apache/spark/pull/52637#discussion_r2438357910) and
the rounding mode itself was not discussed. It shipped in 4.1.0 and is in 4.2.0
and 4.3.0-rc1.
I reproduced this against a distribution built from unmodified master and
against released 4.2.0, in Classic and Connect sessions, with pandas 3.0.5 and
pyarrow 25.0.1. With this change every path returns 1.01, 1.03 and 0.13, the
same as `CAST`.
### Does this PR introduce _any_ user-facing change?
Yes. A decimal returned by an Arrow-optimized Python UDF, an Arrow UDTF or a
Python data source, or passed to `createDataFrame` through Spark Connect, that
has more fractional digits than the declared scale and sits exactly on a tie
now rounds HALF_UP like `CAST` does, instead of HALF_EVEN. Values that are not
on a tie are unchanged. A migration guide entry is included; since this is a
wrong-result fix in released 4.1.x and 4.2.x, I would propose backporting it to
the maintenance branches, and I can move the note to whichever section the fix
version needs.
### How was this patch tested?
New tests: `test_decimal_rescale_rounds_half_up` in `test_conversion.py`;
`test_decimal_round_half_up` in `test_arrow_python_udf.py` (three Classic
suites and their three Connect parity suites), `test_udtf.py` (Classic and
Connect parity), `test_python_datasource.py` (Classic and Connect parity) and
`test_creation.py` (Classic and Connect parity). On master every one of them
fails with the HALF_EVEN values except the Classic `createDataFrame` test,
which passes on master because Classic rescales on the JVM and stays green. The
modules `test_conversion`, `test_arrow_python_udf`, `test_creation`,
`test_parity_creation`, `test_udf`, `test_udtf` and `test_python_datasource`
pass with the change (the two `test_udf_with_input_file_name` tests in
`test_udf` and `test_arrow_python_udf` read a path relative to the source tree
and were run from there). The reproduction script from the ticket was run
against released 4.2.0 and against a master build with and without the change,
in Classic an
d Connect sessions.
### Was this patch authored or co-authored using generative AI tooling?
Authored by Li Guo, assisted by Claude Code (Fable 5.1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
