jubins opened a new pull request, #59214: URL: https://github.com/apache/spark/pull/59214
### What changes were proposed in this pull request? Fixes [SPARK-59910](https://issues.apache.org/jira/browse/SPARK-59910). `VariantVal.toJson()` (and `str(VariantVal)`) is implemented in pure Python in `pyspark/sql/variant_utils.py`. It formatted scalars with Python's `str()`/`repr()`, which differs from the JVM implementation (`Variant.toJsonImpl` in `common/variant`). For non-finite doubles and floats it produced output that is not valid JSON. This PR formats scalars the same way as the JVM: - **Doubles and floats**: NaN and infinities are quoted (`"NaN"`, `"Infinity"`, `"-Infinity"`), as in the JVM. Finite values are formatted like Java's `Double.toString` / `Float.toString`: - the shortest correctly rounded decimal that round-trips, with at least 2 significant digits; - plain notation for magnitudes in [1e-3, 1e7), and `E` notation otherwise. A 4-byte FLOAT is formatted using float32 precision rather than as the widened double. - **Decimals**: formatted like `BigDecimal.stripTrailingZeros().toPlainString()`, with no trailing zeros and never scientific notation. - **Timestamps**: the fractional seconds have no trailing zeros, and the UTC offset is printed as `+HH:MM`, dropping the seconds of historical local-mean-time offsets, as `DateTimeFormatter` does. `toPython()` is unchanged. **Brief change log** - `python/pyspark/sql/variant_utils.py`: adds `_float_to_json`, `_decimal_to_json` and `_timestamp_to_json`, and uses them in `_to_json`. - `python/pyspark/sql/tests/test_types.py`: - Adds `test_variant_to_json_matches_jvm_format`, which builds variant bytes directly and checks each formatting rule and that the output parses as JSON. - Updates `test_variant_type` expectations that had pinned down the old output. ### Why are the changes needed? `toJson()` can return invalid JSON, and it returns different text than `to_json` for the same variant: ```python spark.conf.set("spark.sql.session.timeZone", "UTC") e = "cast(double('NaN') as variant)" r = spark.sql(f"select {e} v, to_json({e}) j").first() r.j # '"NaN"' r.v.toJson() # 'nan' -> json.loads() fails ``` | Value | JVM `to_json` | Before | After | |---|---|---|---| | double NaN | `"NaN"` | `nan` (invalid JSON) | `"NaN"` | | double Infinity | `"Infinity"` | `inf` (invalid JSON) | `"Infinity"` | | double 1e7 | `1.0E7` | `10000000.0` | `1.0E7` | | float 1.1 | `1.1` | `1.100000023841858` | `1.1` | | decimal(5,2) 1.00 | `1` | `1.00` | `1` | | decimal(10,9) 0.000000001 | `0.000000001` | `1E-9` | `0.000000001` | | timestamp_ntz `1970-01-01 00:00:01.5` | `"1970-01-01 00:00:01.5"` | `"1970-01-01 00:00:01.500000"` | `"1970-01-01 00:00:01.5"` | | timestamp `1800-01-01 00:00:00` (America/Los_Angeles) | `"...-07:52"` | `"...-07:52:58"` | `"...-07:52"` | ### Does this PR introduce _any_ user-facing change? Yes. The strings returned by `VariantVal.toJson()` and `str(VariantVal)` change for doubles, floats, decimals and timestamps, as shown in the table above. They now match what the `to_json` SQL function returns for the same value, and non-finite numbers no longer produce invalid JSON. The behaviour has been in place since `VariantVal` was added in 4.0.0. ### How was this patch tested? - `DataTypeTests.test_variant_to_json_matches_jvm_format` (new, no JVM needed). 15 of its cases fail without this change. - `TypesTests.test_variant_type` (updated expectations). - A differential check against the JVM, not included in the PR: 4,623 variant values in each of 4 session time zones (UTC, America/Los_Angeles, Asia/Kolkata, Asia/Kathmandu), 18,492 comparisons of `VariantVal.toJson()` with the `to_json` result for the same value. The values included: - 2,000 random doubles and 2,000 random floats (including subnormals and extremes); - 300 random decimals of up to 28 digits; - 300 random timestamps across years 1 to 9999, plus nested arrays and objects. All values matched, except a 29-digit decimal. That value is affected by a separate precision issue in `_get_decimal`: it rounds to Python's default 28-digit decimal context, which also affects `toPython()`. That issue is out of scope for this PR. The check ran on JDK 21. The float formatting follows the `Double.toString` specification of JDK 19 and later; earlier JDKs can print a longer digit string for a small number of values. ``` python -m pytest pyspark/sql/tests/test_types.py pyspark/sql/tests/connect/test_parity_types.py ``` 218 passed. `TypesParityTests.test_infer_schema_specification` fails, but it fails the same way on master without this change (a `TIME` type error unrelated to variants). `ruff check`, `ruff format --check` and `mypy` pass. ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes - Claude Code was used as a pair-programming assistant. All changes were reviewed and verified by the author. Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
