This is an automated email from the ASF dual-hosted git repository.
nchammas pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new ba57d30eaa30 [SPARK-58282][PYTHON][DOC] Refresh PySpark README and
clarify project scope
ba57d30eaa30 is described below
commit ba57d30eaa305cd1b0cde969a12987301c1e6e05
Author: Nicholas Chammas <[email protected]>
AuthorDate: Wed Aug 19 10:40:58 2026 -0400
[SPARK-58282][PYTHON][DOC] Refresh PySpark README and clarify project scope
### What changes were proposed in this pull request?
Refresh the PySpark README.
Key changes:
- Remove the "experimental" label for the Python distribution of Spark.
- Add a summary of the different packages on PyPI and scope claims about
JARs being included accordingly.
- Use the term "distribution" more deliberately and clarify Python vs.
"Full" distribution.
### Why are the changes needed?
The README hasn't seen a significant update since it was introduced in
#15659 about 10 years ago, and contains some obsolete language.
Clarifying the intended use of PySpark vs. "Full" Spark will also help
focus related discussions coming out of the effort to potentially introduce a
new CLI in #57313.
### Does this PR introduce _any_ user-facing change?
Documentation only.
### How was this patch tested?
No testing.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57452 from nchammas/pyspark-readme.
Authored-by: Nicholas Chammas <[email protected]>
Signed-off-by: Nicholas Chammas <[email protected]>
(cherry picked from commit 75ac74bafec1f489224765041a19bec6bad36a9d)
Signed-off-by: Nicholas Chammas <[email protected]>
---
python/README.md | 33 ++++++++++++++++++---------------
1 file changed, 18 insertions(+), 15 deletions(-)
diff --git a/python/README.md b/python/README.md
index ca4e493fdd39..9fc7cf54f510 100644
--- a/python/README.md
+++ b/python/README.md
@@ -1,33 +1,36 @@
# Apache Spark
Spark is a unified analytics engine for large-scale data processing. It
provides
-high-level APIs in Scala, Java, Python, and R, and an optimized engine that
+high-level APIs in Scala, Java, Python, and R (deprecated), and an optimized
engine that
supports general computation graphs for data analysis. It also supports a
rich set of higher-level tools including Spark SQL for SQL and DataFrames,
pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX
for graph processing,
and Structured Streaming for stream processing.
-<https://spark.apache.org/>
+PySpark is the Python distribution of Spark.
-## Online Documentation
+Project home page: https://spark.apache.org/
-You can find the latest Spark documentation, including a programming
-guide, on the [project web page](https://spark.apache.org/documentation.html)
+Main documentation: https://spark.apache.org/docs/latest/
+PySpark documentation:
https://spark.apache.org/docs/latest/api/python/index.html
-## Python Packaging
+## PySpark on PyPI
-This README file only contains basic information related to pip installed
PySpark.
-This packaging is currently experimental and may change in future versions
(although we will do our best to keep compatibility).
-Using PySpark requires the Spark JARs, and if you are building this from
source please see the builder instructions at
-["Building Spark"](https://spark.apache.org/docs/latest/building-spark.html).
+There are a few PySpark packages published by the Apache Spark project to PyPI:
-The Python packaging for Spark is not intended to replace all of the other use
cases. This Python packaged version of Spark is suitable for interacting with
an existing cluster (be it Spark standalone, YARN) - but does not contain the
tools required to set up your own standalone Spark cluster. You can download
the full version of Spark from the [Apache Spark downloads
page](https://spark.apache.org/downloads.html).
+- `pyspark`: Classic PySpark, includes Spark assembly JARs
+- `pyspark-connect`: Classic PySpark with Spark Connect configured as the
default, includes `pyspark`
+- `pyspark-client`: Pure Python Spark Connect client, no JARs or JRE needed
+For more information, see the [installation guide][install]. If you're
building PySpark from source, see [Building Spark][build] and
[pyproject.toml][py].
-**NOTE:** If you are using this with a Spark standalone cluster you must
ensure that the version (including minor version) matches or you may experience
odd errors.
+[install]:
https://spark.apache.org/docs/latest/api/python/getting_started/install.html
+[build]: https://spark.apache.org/docs/latest/building-spark.html
+[py]: https://github.com/apache/spark/blob/master/pyproject.toml
-## Python Requirements
+## Python vs. "Full" Distribution of Spark
-At its core PySpark depends on Py4J, but some additional sub-packages have
their own extra requirements for some features (including numpy, pandas, and
pyarrow).
-See also
[Dependencies](https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies)
for production, and
[pyproject.toml](https://github.com/apache/spark/blob/master/pyproject.toml)
for development.
+PySpark is not intended to be a complete distribution of Spark. It's meant for
local development or for interacting with an existing cluster (be it Spark
standalone, YARN, or Kubernetes). Using PySpark to set up a new standalone
Spark cluster is not supported. To set up a standalone cluster please [use the
full distribution of Spark](https://spark.apache.org/downloads.html).
+
+When using PySpark with an existing Spark cluster you must ensure that the
major and minor version (e.g. `4.3.*`) match or you may experience odd errors.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
