HyukjinKwon opened a new pull request, #57068: URL: https://github.com/apache/spark/pull/57068
### What changes were proposed in this pull request? This PR makes the PySpark Spark Connect client raise a clean `PySparkNotImplementedError` (error class `NOT_IMPLEMENTED`) when a result containing a `YearMonthIntervalType` is collected, instead of surfacing an opaque PyArrow error. Concretely: - `from_arrow_type` (in `pyspark/sql/pandas/types.py`) now maps the Arrow `YEAR_MONTH` interval type to `YearMonthIntervalType`. The JVM serializes Spark's `YearMonthIntervalType` to an Arrow `YEAR_MONTH` interval, but PyArrow exposes no `is_*()` helper or factory for it (only `MONTH_DAY_NANO` is in `pyarrow.types`), so the branch matches on the stable Arrow type id (`Type::INTERVAL_MONTHS == 21`). - `ArrowTableToRowsConversion.convert` (in `pyspark/sql/conversion.py`) checks the result schema before materializing and raises `NOT_IMPLEMENTED` if any field (including nested array/map/struct/UDT element types) is a `YearMonthIntervalType`. PyArrow cannot materialize such a column: `to_pylist()` raises an opaque `KeyError: 21` from `get_array_class_from_type`, and that lookup fails even for an empty column, so the check is intentionally unconditional in the row count. - `_has_type` (in `pyspark/sql/types.py`) now recurses into `UserDefinedType.sqlType()` so a year-month interval hidden inside a UDT is detected too. This matches the default behavior of the classic PySpark path, where `YearMonthIntervalType.fromInternal` raises `NOT_IMPLEMENTED`. ### Why are the changes needed? Collecting a year-month interval through the Spark Connect client previously failed with confusing, low-level errors rather than the intended `NOT_IMPLEMENTED`: - `PySparkTypeError: [UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION] month_interval is not supported`, or - `pyarrow.lib.ArrowNotImplementedError: No known equivalent Pandas block for Arrow data of type month_interval`, or - an opaque `KeyError: 21` from PyArrow when materializing rows. Classic PySpark already raises a clear `NOT_IMPLEMENTED` error for this unsupported operation (`YearMonthIntervalType.fromInternal`). Spark Connect should behave the same so users get an actionable message and the two clients stay consistent. ### Does this PR introduce _any_ user-facing change? Yes, for the Spark Connect Python client. Collecting a year-month interval value (`df.collect()`/`first()`/`take()`/`head()`) now raises `PySparkNotImplementedError` with error class `NOT_IMPLEMENTED` instead of a `PySparkTypeError`/`ArrowNotImplementedError`/`KeyError`. Collecting a year-month interval was never supported; only the surfaced error changes. Two behaviors remain specific to Spark Connect and differ from classic PySpark: - `PYSPARK_YM_INTERVAL_LEGACY=1` (which makes classic return the internal integer months) is not honored; collect always raises. - An empty result (e.g. `.limit(0).collect()`) raises rather than returning `[]`, because PyArrow cannot build the `INTERVAL_MONTHS` array class regardless of row count. ### How was this patch tested? - Updated `test_connect_error.SparkConnectErrorTests.test_ym_interval_in_collect` to expect `PySparkNotImplementedError` and added coverage for a year-month interval nested inside an array. - Added `test_connect_error.SparkConnectErrorTests.test_ym_interval_empty_collect` covering the empty-result case. - Updated the skip reason on `test_parity_types` for `test_ym_interval_in_collect` to explain why the inherited classic contract (which asserts the `PYSPARK_YM_INTERVAL_LEGACY=1` integer-months behavior) cannot be satisfied by Spark Connect. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
