This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 41dfd701c8aa [SPARK-57974][PYTHON][DOC] Add pandas upper bound
`<3.0.0` in `dev/requirements.txt` and `install.rst`
41dfd701c8aa is described below
commit 41dfd701c8aabdb0e9ac2dd832caf24a8aaa592c
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Jul 6 14:18:26 2026 -0700
[SPARK-57974][PYTHON][DOC] Add pandas upper bound `<3.0.0` in
`dev/requirements.txt` and `install.rst`
This PR aims to add an upper bound `<3.0.0` to the `pandas` requirement in
`dev/requirements.txt` and the installation guide
(`python/docs/source/getting_started/install.rst`) because Apache Spark 4.2.0
doesn't support it. Apache Spark 4.3.0 will support `Pandas 3` via SPARK-55139
officially.
PySpark does not yet fully support pandas 3.x and warns at runtime when
pandas >= 3.0.0 is detected:
https://github.com/apache/spark/blob/4553429c621044bb30d20b901f61bed8b2089817/python/pyspark/sql/pandas/utils.py#L63-L65
No. This only affects the development requirements file and documentation.
Manual review.
Generated-by: Claude Fable 5
Closes #57051 from dongjoon-hyun/SPARK-57974.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 3e88122235c95d671edd3019025e6929c7c3ca81)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/requirements.txt | 2 +-
python/docs/source/getting_started/install.rst | 8 ++++----
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/dev/requirements.txt b/dev/requirements.txt
index cde0957715bf..f9082acd04ef 100644
--- a/dev/requirements.txt
+++ b/dev/requirements.txt
@@ -5,7 +5,7 @@ py4j>=0.10.9.9
numpy>=1.22
pyarrow>=15.0.0
six==1.16.0
-pandas>=2.2.0
+pandas>=2.2.0,<3.0.0
scipy
plotly<6.0.0
mlflow>=2.3.1
diff --git a/python/docs/source/getting_started/install.rst
b/python/docs/source/getting_started/install.rst
index 6b5a09205e4a..961ba5492760 100644
--- a/python/docs/source/getting_started/install.rst
+++ b/python/docs/source/getting_started/install.rst
@@ -225,7 +225,7 @@ Installable with ``pip install "pyspark[connect]"``.
========================== ================= ==========================
Package Supported version Note
========================== ================= ==========================
-`pandas` >=2.2.0 Required for Spark Connect
+`pandas` >=2.2.0,<3.0.0 Required for Spark Connect
`pyarrow` >=15.0.0 Required for Spark Connect
`grpcio` >=1.76.0 Required for Spark Connect
`grpcio-status` >=1.76.0 Required for Spark Connect
@@ -242,7 +242,7 @@ Installable with ``pip install "pyspark[sql]"``.
========= ================= ======================
Package Supported version Note
========= ================= ======================
-`pandas` >=2.2.0 Required for Spark SQL
+`pandas` >=2.2.0,<3.0.0 Required for Spark SQL
`pyarrow` >=15.0.0 Required for Spark SQL
========= ================= ======================
@@ -259,7 +259,7 @@ Installable with ``pip install "pyspark[pandas_on_spark]"``.
========= ================= ================================
Package Supported version Note
========= ================= ================================
-`pandas` >=2.2.0 Required for Pandas API on Spark
+`pandas` >=2.2.0,<3.0.0 Required for Pandas API on Spark
`pyarrow` >=15.0.0 Required for Pandas API on Spark
========= ================= ================================
@@ -309,7 +309,7 @@ Installable with ``pip install "pyspark[pipelines]"``.
Includes all dependencies
========================== =================
===================================================
Package Supported version Note
========================== =================
===================================================
-`pandas` >=2.2.0 Required for Spark Connect and
Spark SQL
+`pandas` >=2.2.0,<3.0.0 Required for Spark Connect and
Spark SQL
`pyarrow` >=15.0.0 Required for Spark Connect and
Spark SQL
`grpcio` >=1.76.0 Required for Spark Connect
`grpcio-status` >=1.76.0 Required for Spark Connect
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
