Spenserrrr opened a new pull request, #58537: URL: https://github.com/apache/spark/pull/58537
### What changes were proposed in this pull request? Add golden-file tests for the pandas type-introspection predicates (`pandas.api.types.is_*`) that PySpark depends on, in a new `python/pyspark/tests/upstream/pandas` package. Two classes split by input axis, matching how pandas itself implements them: the dtype predicates take a dtype or an array, the object predicates take an arbitrary object. ### Why are the changes needed? PySpark branches on the boolean these predicates return, so an upstream change misroutes control flow instead of corrupting one value. For example `_create_from_pandas_with_arrow` maps a pandas column to `TimestampType` only when `is_datetime64_dtype` says so, and that predicate answers `False` for a tz-aware or arrow-backed timestamp dtype. Nothing currently pins those answers. ### Does this PR introduce _any_ user-facing change? No. Tests only. ### How was this patch tested? New tests, run across PyArrow 18-25 x pandas 2/3 (16/16 pass); the four cells that move on pandas 3 are recorded as version-guarded `overrides`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
