cloud-fan opened a new pull request, #56553: URL: https://github.com/apache/spark/pull/56553
### What changes were proposed in this pull request? An audit of Spark 4.2 commits (SPARK-57452) found a number of user-facing behavior changes that were not documented in the migration guides. This PR adds entries for the subset of those changes that **require a user to take action to migrate**, across three guides: PySpark (`python/docs/source/migration_guide/pyspark_upgrade.rst`): - PyPy is no longer officially supported. - Minimum pandas version for Spark Connect raised to 2.2.0. - `createDataFrame` from a NumPy `ndarray` now requires PyArrow and may infer a different schema. - Pandas UDFs receive nullable integer columns as a pandas nullable `Int` dtype instead of `float64`. - `Observation.get` raises on metric collection failure instead of returning an empty dictionary. - `DataFrame.drop` / `Series.drop` (pandas API on Spark) raise `KeyError` if any label is missing. - Python Data Sources fail with `DATA_SOURCE_RETURN_SCHEMA_MISMATCH` on schema mismatch. - Python streaming data sources fail when reported offsets do not advance. SQL (`docs/sql-migration-guide.md`): - Duplicate CTE names within a `WITH` clause are detected case-insensitively. - `NATURAL JOIN` honors `spark.sql.caseSensitive` when choosing join columns. - Built-in parameterless functions take precedence over same-named UDFs. - `SET CATALOG <name>` resolves a bare name as a session variable first. Structured Streaming (`docs/streaming/ss-migration-guide.md`): - Restarting from a checkpoint with missing required metadata now fails by default. Behavior changes that are purely cosmetic/UI, internal, additive, or correctness fixes requiring no user action were intentionally excluded. ### Why are the changes needed? Without these entries, users upgrading to Spark 4.2 can hit silent result changes, new errors on previously-valid input, or dependency/environment requirements with no documented guidance on how to migrate. ### Does this PR introduce _any_ user-facing change? No. This is a documentation-only change. ### How was this patch tested? Documentation-only change; verified by review of the rendered Markdown/reStructuredText. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
