HyukjinKwon commented on code in PR #59214:
URL: https://github.com/apache/spark/pull/59214#discussion_r4169781569
##########
python/pyspark/sql/variant_utils.py:
##########
@@ -413,10 +417,81 @@ def handle_array(value_pos_list: List[int]) -> str:
if isinstance(value, bytes):
# decoding simply converts byte array to string
return '"' + base64.b64encode(value).decode("utf-8") + '"'
- if isinstance(value, (datetime.date, datetime.datetime)):
+ if isinstance(value, float):
+ return cls._float_to_json(value, is_float32)
+ if isinstance(value, decimal.Decimal):
+ return cls._decimal_to_json(value)
+ if isinstance(value, datetime.datetime):
+ return '"' + cls._timestamp_to_json(value) + '"'
+ if isinstance(value, datetime.date):
return '"' + str(value) + '"'
return str(value)
+ # The helpers below format scalars the same way as the JVM
`Variant.toJson`, so that
+ # `VariantVal.toJson` and the `to_json` SQL function produce the same
output.
+
+ @classmethod
+ def _float_to_json(cls, f: float, is_float32: bool) -> str:
+ """
+ Formats a double or float like Java's `Double.toString` and
`Float.toString`. Non-finite
+ values are quoted because they are not valid JSON numbers.
+ """
+ if math.isnan(f):
+ return '"NaN"'
+ if math.isinf(f):
+ return '"Infinity"' if f > 0 else '"-Infinity"'
+
+ def round_trips(s: str) -> bool:
+ if not is_float32:
+ return float(s) == f
+ try:
+ return struct.pack("<f", float(s)) == struct.pack("<f", f)
+ except OverflowError:
+ return False
+
+ # Like Java, pick the shortest correctly rounded decimal that
round-trips, using at least
+ # 2 significant digits (e.g. Double.MIN_VALUE is "4.9E-324", not
"5.0E-324").
+ for precision in range(2, 18):
Review Comment:
**Nit (P3):** This only tries the correctly rounded `precision`-digit
decimal. When `f` is a power of two, the rounding interval below `f` is half as
wide as the one above it. So the nearest n-digit decimal can fail to round-trip
while the next n-digit decimal away from zero still does, and JDK 19+
`Double.toString`/`Float.toString` pick that one.
I checked against Temurin JDK 25 using every double and float power of two
plus 200k random bit patterns of each. The random values all match, but 46
double powers of two and 3 float powers of two come out one digit longer than
the JVM:
| value | JVM `to_json` | this PR |
|---|---|---|
| double 2^-24 | `5.960464477539063E-8` | `5.9604644775390625E-8` |
| double 2^-44 | `5.684341886080802E-14` | `5.6843418860808015E-14` |
| float 2^-96 | `1.2621775E-29` | `1.26217745E-29` |
(Before this PR, `str()`/repr already printed the 16-digit form for these
doubles.) The output still parses to the same value, so only parity with
`to_json` is affected. When the rounded candidate fails, also trying the
decimal one unit in the last place away from zero at the same precision before
increasing `precision` fixed all 49 cases, with no other output changes across
the 401,418 values checked.
**Verification:**
- **Regression:** Assert that power-of-two doubles and floats whose shortest
decimal lies above the nearest rounding (e.g. 2^-24 double, 2^-96 float) format
to the JDK 19+ strings.
- **Compatibility:** Existing JVM-format and variant type expectations
(NaN/Infinity, signed zero, plain/E boundaries, Double.MIN_VALUE,
Float.MAX_VALUE, decimals, timestamps) still pass unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
