Akshay Thorat created SPARK-59515:
-------------------------------------
Summary: assertSchemaEqual ignores non-nullability type parameters
Key: SPARK-59515
URL: https://issues.apache.org/jira/browse/SPARK-59515
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 5.0.0
Reporter: Akshay Thorat
h2. Problem
With ignoreNullable=True, assertSchemaEqual accepts remaining leaf types solely
because typeName matches. This ignores semantic parameters unrelated to
nullability, including collation, character length, time precision, and
interval bounds.
h2. Reproduction
Reproduced on upstream master 39776477a3d (PySpark 5.0.0.dev0), Python 3.10.11.
No SparkSession is required.
{code:python}
from pyspark.sql.types import StringType, StructField, StructType
from pyspark.testing import assertSchemaEqual
a = StructType([StructField("s", StringType())])
b = StructType([StructField("s", StringType("UTF8_LCASE"))])
assertSchemaEqual(a, b) # Returns without raising
{code}
h2. Expected behavior
Different collation semantics should raise DIFFERENT_SCHEMA even when
nullability is ignored. The same applies to different CharType/VarcharType
lengths, TimeType precision, and interval bounds.
h2. Proposed fix and verification
Use data-type equality for leaf types after the existing recursive nullability
handling. Regression tests cover seven type pairs across scalar, array, map
key, map value, and nested struct contexts. Nested nullability continues to be
ignored. User tests relying on the previous loose matching may now fail
correctly with DIFFERENT_SCHEMA.
Regression tests fail before the fix and pass afterward. Python compilation,
custom-error checks and Ruff checks pass.
Pull request: https://github.com/apache/spark/pull/58791
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
