[
https://issues.apache.org/jira/browse/SPARK-59910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59910:
-----------------------------------
Labels: pull-request-available (was: )
> VariantVal.toJson in PySpark produces invalid JSON for NaN/Infinity and
> diverges from JVM to_json output
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59910
> URL: https://issues.apache.org/jira/browse/SPARK-59910
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.1.0, 4.0.0, 4.2.0
> Reporter: Jubin Soni
> Priority: Major
> Labels: pull-request-available
>
> The pure-Python variant decoder (pyspark/sql/variant_utils.py) used by
> VariantVal.toJson() formats scalars differently from the JVM implementation
> (Variant.toJsonImpl in common/variant). For non-finite doubles and floats it
> produces output that is not valid JSON.
> h3. Reproduction
> {code:python}
> spark.conf.set("spark.sql.session.timeZone", "UTC")
> e = "cast(double('NaN') as variant)"
> r = spark.sql(f"select \{e} v, to_json(\{e}) j").first()
> r.j # '"NaN"'
> r.v.toJson() # 'nan' -> json.loads() fails
> {code}
> h3. Python vs JVM output
> ||Value||JVM to_json||Python VariantVal.toJson()||
> |double NaN|"NaN"|nan (invalid JSON)|
> |double Infinity|"Infinity"|inf (invalid JSON)|
> |float 1.1|1.1|1.100000023841858|
> |decimal(5,2) 1.00|1|1.00|
> |decimal(10,9) 0.000000001|0.000000001|1E-9|
> |timestamp_ntz 1970-01-01 00:00:01.5|"1970-01-01 00:00:01.5"|"1970-01-01
> 00:00:01.500000"|
> |timestamp 1970-01-01 00:00:01.5 (UTC)|"1970-01-01
> 00:00:01.5+00:00"|"1970-01-01 00:00:01.500000+00:00"|
> h3. Proposed fix
> Match the JVM formatting in the Python toJson path:
> * quote non-finite doubles and floats
> * print FLOAT with its shortest float32 representation
> * print decimals with trailing zeros stripped, in plain (non-scientific)
> notation
> * print the fractional seconds of timestamps without trailing zeros
> toPython() behaviour is unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
