Jubin Soni created SPARK-59910:
----------------------------------
Summary: VariantVal.toJson in PySpark produces invalid JSON for
NaN/Infinity and diverges from JVM to_json output
Key: SPARK-59910
URL: https://issues.apache.org/jira/browse/SPARK-59910
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 4.2.0, 4.0.0, 4.1.0
Reporter: Jubin Soni
The pure-Python variant decoder (pyspark/sql/variant_utils.py) used by
VariantVal.toJson() formats scalars differently from the JVM implementation
(Variant.toJsonImpl in common/variant). For non-finite doubles and floats it
produces output that is not valid JSON.
h3. Reproduction
{code:python}
spark.conf.set("spark.sql.session.timeZone", "UTC")
e = "cast(double('NaN') as variant)"
r = spark.sql(f"select \{e} v, to_json(\{e}) j").first()
r.j # '"NaN"'
r.v.toJson() # 'nan' -> json.loads() fails
{code}
h3. Python vs JVM output
||Value||JVM to_json||Python VariantVal.toJson()||
|double NaN|"NaN"|nan (invalid JSON)|
|double Infinity|"Infinity"|inf (invalid JSON)|
|float 1.1|1.1|1.100000023841858|
|decimal(5,2) 1.00|1|1.00|
|decimal(10,9) 0.000000001|0.000000001|1E-9|
|timestamp_ntz 1970-01-01 00:00:01.5|"1970-01-01 00:00:01.5"|"1970-01-01
00:00:01.500000"|
|timestamp 1970-01-01 00:00:01.5 (UTC)|"1970-01-01
00:00:01.5+00:00"|"1970-01-01 00:00:01.500000+00:00"|
h3. Proposed fix
Match the JVM formatting in the Python toJson path:
* quote non-finite doubles and floats
* print FLOAT with its shortest float32 representation
* print decimals with trailing zeros stripped, in plain (non-scientific)
notation
* print the fractional seconds of timestamps without trailing zeros
toPython() behaviour is unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
