Akshay Thorat created SPARK-59512:
-------------------------------------
Summary: StructField.fromJson mutates collation metadata and fails
on repeated parsing
Key: SPARK-59512
URL: https://issues.apache.org/jira/browse/SPARK-59512
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 5.0.0
Reporter: Akshay Thorat
h2. Problem
Parsing a cached schema dictionary changes its collation metadata. A second
StructType.fromJson call on the same dictionary raises AssertionError because
the provider prefix was removed.
h2. Reproduction
Reproduced from upstream master 39776477a3d using repository Python source
(PySpark 5.0.0.dev0), Python 3.10.11 and NumPy 2.2.6. No SparkSession is
required.
{code:python}
from pyspark.sql.types import StringType, StructField, StructType
payload = StructType([StructField("s", StringType("UTF8_LCASE"))]).jsonValue()
StructType.fromJson(payload)
StructType.fromJson(payload) # AssertionError
{code}
h2. Expected behavior
Both calls should return the same schema and leave payload unchanged.
h2. Proposed fix and verification
Copy the collation lookup before stripping provider names in
StructField.fromJson. Regression tests cover repeated parsing with collated
strings, arrays, maps, and nested structs. The new regression tests fail before
the fix and pass afterward. Python compilation, custom-error checks and Ruff
checks pass.
Pull request: https://github.com/apache/spark/pull/58787
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
