Haotian Sun created SPARK-59271:
-----------------------------------
Summary: Add tests for pandas.api.types is_* predicates
Key: SPARK-59271
URL: https://issues.apache.org/jira/browse/SPARK-59271
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 5.0.0
Reporter: Haotian Sun
Add golden-file tests pinning the behavior of pandas' type-introspection
predicates
(pandas.api.types.is_*) that PySpark depends on.
PySpark branches on the boolean these predicates return, so an upstream change
misroutes
control flow rather than corrupting a single value: the conversion layer picks
a Spark type
from is_datetime64_dtype / is_timedelta64_dtype, is_integer_dtype decides
whether integer
data is rewritten as Decimal, and pandas-on-Spark dispatches implementations
and raises
user-visible TypeErrors from is_hashable / is_list_like.
The tests run without a Spark session and act as a drift canary, following the
pattern of
the existing python/pyspark/tests/upstream/pyarrow suite.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
