laserninja opened a new pull request, #58791:
URL: https://github.com/apache/spark/pull/58791
### What changes were proposed in this pull request?
Use data-type equality for leaf types after the existing recursive
nullability handling. Add coverage for collations, character lengths, time
precision, interval bounds, and nested containers.
### Why are the changes needed?
The ignoreNullable comparator accepts remaining types solely because
typeName matches. This also ignores semantic parameters unrelated to
nullability, such as UTF8_BINARY versus UTF8_LCASE collations.
```python
from pyspark.sql.types import StringType, StructField, StructType
from pyspark.testing import assertSchemaEqual
a = StructType([StructField("s", StringType())])
b = StructType([StructField("s", StringType("UTF8_LCASE"))])
assertSchemaEqual(a, b) # Returns without raising
```
### Does this PR introduce _any_ user-facing change?
Yes. With ignoreNullable=True, schemas with unequal leaf-type parameters now
raise DIFFERENT_SCHEMA. Recursive array, map, and struct nullability remains
ignored.
### How was this patch tested?
The new regression tests failed before the fix and passed afterward on
Python 3.10.11 with NumPy 2.2.6. The focused run passed 2 tests, including
parameterized subtests:
```bash
PYTHONPATH=python TERM=dumb python -m unittest
pyspark.sql.tests.test_utils.SchemaComparisonTests -q
PYTHON_EXECUTABLE=python ./dev/lint-python --compile --custom-pyspark-error
--ruff
```
Run the commands with the Python virtual environment activated; the local
run shared an environment from a sibling checkout. The tests call public Python
APIs directly and require no SparkSession. Python compilation, custom-error
checks, Ruff lint, and Ruff formatting passed. JVM-backed suites were not run
because these fixes are confined to Python-side logic.
Draft pending creation and linking of the corresponding ASF Jira issue.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
