Arthur031221 opened a new pull request, #59201:
URL: https://github.com/apache/spark/pull/59201
### What changes were proposed in this pull request?
Decode variant decimals from their coefficient digits and scale without
decimal context arithmetic. Add a regression test for DECIMAL4, DECIMAL8, and
DECIMAL16 values under precision contexts of 1, 28, and 38.
### Why are the changes needed?
Users decoding variant decimals with more than 28 significant digits through
`VariantVal.toPython()` or `toJson()` receive rounded values under the default
Python decimal context because decoding multiplies by a power of ten using
context arithmetic. With the default precision of 28, a stored 38-digit value
loses digits. Enabling the `Inexact` trap can turn decoding into an exception.
### Does this PR introduce _any_ user-facing change?
Yes. Decimal values retain their stored precision when converted to Python
or JSON. The rounding also reproduces with PySpark 4.0.0.
```python
from pyspark.sql.types import VariantVal
value = VariantVal.parseJson("0.12345678901234567890123456789012345678")
print(value.toPython())
```
Before: `0.1234567890123456789012345679`
After: `0.12345678901234567890123456789012345678`
### How was this patch tested?
Added `VariantValTests.test_decimal_precision`. The test fails with the
previous decoder, passes with the fix, and fails again when the decoder is
restored from master. It also checks that decoding leaves decimal context flags
clear.
```sh
PYTHONPATH=python python -m unittest -q \
pyspark.sql.tests.test_types.VariantValTests \
pyspark.sql.tests.test_types.DataTypeTests \
pyspark.sql.tests.test_types.DataTypeVerificationTests \
pyspark.sql.tests.test_conversion
dev/lint-python --ruff
```
75 tests passed on Python 3.12.3. The Ruff checks and formatting checks
passed.
Also tested six decimal values collected from Spark 4.0.0 under the three
precision contexts, with the same decoder change applied to that release. All
18 values matched the JVM JSON values; the original decoder raised `Inexact` in
eight cases. This used the release JARs, without building Spark from this
branch.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex CLI 0.160.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
