Yicong-Huang opened a new pull request, #57094: URL: https://github.com/apache/spark/pull/57094
### What changes were proposed in this pull request? This PR reverts [SPARK-54555](https://issues.apache.org/jira/browse/SPARK-54555) (commit ea0a35e065d, "Enable Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default"), restoring the following config defaults to `false`: - `spark.sql.execution.pythonUDF.arrow.enabled` - `spark.sql.execution.pythonUDTF.arrow.enabled` - `spark.sql.execution.arrow.pyspark.enabled` (via its fallback `spark.sql.execution.arrow.enabled`) The associated user-facing documentation and migration-guide entries added by SPARK-54555 are reverted as well, and the `test_unified_udf` test is restored to no longer force the conf off. The revert was applied with `git revert`; the only manual conflict resolution was in the migration guide and the Arrow/pandas tutorial, where subsequent PRs had touched adjacent lines. Only the entries introduced by SPARK-54555 were removed; unrelated 4.1-to-4.2 migration notes are preserved. ### Why are the changes needed? Enabling these Arrow optimizations by default in 4.2.0 is being reverted so the change does not ship on by default in the 4.2.0 release. Users who want the Arrow-optimized behavior can still opt in explicitly via the same configs. ### Does this PR introduce _any_ user-facing change? Yes. It restores the previous default configuration: regular Python UDFs, Python UDTFs, and PySpark Arrow-based columnar data exchange (`DataFrame.toPandas` / `SparkSession.createDataFrame`) are once again non-Arrow by default. This is a revert of an unreleased change (SPARK-54555 targeted 4.2.0), so there is no behavior change relative to released Spark versions. ### How was this patch tested? Reverting existing changes; relies on existing PySpark test coverage that runs with the Arrow conf both enabled and disabled. `sql/catalyst` compiles. ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
