cloud-fan commented on code in PR #56553: URL: https://github.com/apache/spark/pull/56553#discussion_r3424792805
########## python/docs/source/migration_guide/pyspark_upgrade.rst: ########## @@ -27,6 +27,14 @@ Upgrading from PySpark 4.1 to 4.2 * In Spark 4.2, columnar data exchange between PySpark and the JVM uses Apache Arrow by default. The configuration ``spark.sql.execution.arrow.pyspark.enabled`` now defaults to true. To restore the legacy (non-Arrow) row-based data exchange, set ``spark.sql.execution.arrow.pyspark.enabled`` to ``false``. * In Spark 4.2, regular Python UDFs are Arrow-optimized by default. The configuration ``spark.sql.execution.pythonUDF.arrow.enabled`` now defaults to true. To restore the legacy behavior for Python UDF execution, set ``spark.sql.execution.pythonUDF.arrow.enabled`` to ``false``. * In Spark 4.2, regular Python UDTFs are Arrow-optimized by default. The configuration ``spark.sql.execution.pythonUDTF.arrow.enabled`` now defaults to true. To restore the legacy behavior for Python UDTF execution, set ``spark.sql.execution.pythonUDTF.arrow.enabled`` to ``false``. +* In Spark 4.2, PyPy is no longer officially supported. Run PySpark on CPython instead. +* In Spark 4.2, the minimum supported version of pandas for Spark Connect has been raised to 2.2.0. Upgrade pandas to 2.2.0 or later. Review Comment: do we need this or pyspark already have a doc page for version bumps? cc @HyukjinKwon @gaogaotiantian -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
