AMC-hawk opened a new pull request, #59212:
URL: https://github.com/apache/spark/pull/59212
### What changes were proposed in this pull request?
`VariantVal.toJson()` decodes variants in pure Python, and its output had
drifted from the JVM's `to_json`. This makes the Python side format scalars the
same way `Variant.toJsonImpl` does:
- NaN and infinity are written as quoted strings (`"NaN"`, `"Infinity"`,
`"-Infinity"`) instead of `nan` / `inf`
- FLOAT values print their shortest float32 form (`1.1`, not
`1.100000023841858`)
- decimals drop trailing zeros and never use scientific notation (`1`, not
`1.00`; `0.000000001`, not `1E-9`)
- timestamps drop trailing zeros in the fraction (`00:00:01.5`, not
`00:00:01.500000`), and offsets are printed as `+HH:MM` like the JVM, which
matters for historical offsets that have seconds
`toPython()` is unchanged.
### Why are the changes needed?
For NaN and infinity the output wasn't valid JSON, so
`json.loads(v.toJson())` failed:
```python
e = "cast(double('NaN') as variant)"
r = spark.sql(f"select {e} v, to_json({e}) j").first()
r.j # '"NaN"'
r.v.toJson() # 'nan'
```
The other cases were valid JSON but didn't match what `to_json` returns for
the same value.
### Does this PR introduce _any_ user-facing change?
Yes, `VariantVal.toJson()` and `str(VariantVal)` now return the same text as
`to_json` for the cases above. One thing I left alone on purpose: exponent
style for doubles (`1e+20` vs the JVM's `1.0E20`). Both are valid JSON with the
same value, and existing tests rely on the Python style.
### How was this patch tested?
Added `test_variant_to_json_matches_jvm`, which compares `toJson()` with
`to_json` for the same expression across three session time zones, so the JVM
output is the expected value. Updated three existing timestamp assertions in
`test_variant_type`. Ran the variant tests in `test_types`, `test_udf`,
`test_udtf`, `test_conversion` and the Arrow/pandas UDF suites locally.
Note: #59201 touches the same file. There's no code conflict, but its new
test expects `toJson()` to return `1E-38` for a tiny decimal, which becomes
plain notation here, so whichever lands second needs a one-line test update.
### Was this patch authored or co-authored using generative AI tooling?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
