This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 3005717da8c6 [SPARK-60855][CORE] Remove the Spark JAR from the Python
worker PYTHONPATH
3005717da8c6 is described below
commit 3005717da8c67fdff7b193e6644a57b6d46a2152
Author: Tian Gao <[email protected]>
AuthorDate: Thu Nov 5 10:42:44 2026 -0800
[SPARK-60855][CORE] Remove the Spark JAR from the Python worker PYTHONPATH
### What changes were proposed in this pull request?
Remove `SparkContext.jarOfObject(this)` from `PythonUtils.sparkPythonPath`,
so the JAR containing Spark's classes (normally the spark-core JAR) is no
longer put on the `PYTHONPATH` of Python processes Spark launches (Python
workers via `PythonWorkerFactory`, `PythonRunner`,
`PythonUDFWorkerSpecification`), nor on the `sys.path` of the embedded
interpreter `InProcessPythonRuntime` sets up. `sparkPythonPath` now contains
only `$SPARK_HOME/python/lib/pyspark.zip` and the py4j zip, or not [...]
The comments and docs that described the JAR entry are updated too: the
`sparkPythonPath` scaladoc, `PythonRunner`, `PythonWorkerEnvironment`,
`test_python_worker_env.py`, `docs/configuration.md`, and the stale
`PySparkAssembly` comment in `project/SparkBuild.scala`. A core migration guide
entry is added for the one case that can notice the change (see below), and a
regression test in `PythonRunnerSuite` guards against the jar entry coming back.
### Why are the changes needed?
**The JAR has not contained any Python code since Spark 1.4.** Here is how
the entry got there and why it is dead:
1. **SPARK-1004 (ff5be9a41e5, April 2014), PySpark on YARN.** To run
PySpark on YARN executors, which have no `SPARK_HOME`, `core/pom.xml` (and
`mllib/pom.xml`) copied `pyspark/*.py` and the unzipped py4j sources into the
JAR as resources. Python can import from a JAR because a JAR is a zip file
(`zipimport`).
2. **SPARK-1549 (951a5d93986, May 2014), Python support in spark-submit.**
This introduced `PythonUtils.sparkPythonPath`, documented as "Get the
PYTHONPATH for PySpark, either from SPARK_HOME, if it is set, or from our JAR".
The commit message says: "we assume we can always find the Python files either
from the Spark assembly JAR ... or from SPARK_HOME". `jarOfObject(this)` (the
JAR containing `PythonUtils`) was added to the path for that reason. SPARK-3008
(2014, "zipimport not able [...]
3. **SPARK-6869 (ebff7327af5, May 2015).** YARN started shipping
`pyspark.zip` and the py4j zip to containers as separate archives and putting
them on the executors' `PYTHONPATH` (today's `Client.findPySparkArchives` /
`PYSPARK_ARCHIVES_PATH`).
4. **SPARK-7485 (82e890fb19d, May 2015), "Remove pyspark files from
assembly".** This removed the Maven resource rules and the py4j unzip step, and
made the SBT `PySparkAssembly` generator build only `python/lib/pyspark.zip`
(it returns `Seq.empty`, so nothing goes into the JAR). The `sparkPythonPath`
line was not updated, and the follow-up 98478752662 only changed the
`SPARK_HOME` entry to `pyspark.zip`.
Since then, no Spark build has packaged Python into a JAR. On current
master, `assembly/pom.xml` and `project/SparkBuild.scala` build `pyspark.zip`
only, and no `.py` file exists under any `src/main/resources`. The published
`spark-core_2.13-4.0.0.jar` has 5,261 entries and no `.py`/`.pyc` files. Every
deployment gets PySpark some other way:
| Deployment | Where `pyspark` is imported from |
|---|---|
| Local / standalone / Kubernetes (distribution or image) |
`$SPARK_HOME/python/lib/pyspark.zip` (this same method) |
| YARN | `pyspark.zip` and py4j shipped by `Client` and set in the
containers' `PYTHONPATH` |
| pip-installed PySpark | site-packages |
So with any Spark-built JAR, the entry can never resolve an import (the one
exception, an application uber-jar with Spark shaded in, is covered below). It
only adds work:
- Python creates a `zipimporter` for the JAR and reads its central
directory (one entry per class file).
- `PYTHONPATH` entries come before the standard library on `sys.path`, so
every top-level import that misses `pyspark.zip` and py4j also probes the JAR.
- Python workers call `importlib.invalidate_caches()` in
`worker_util.setup_spark_files` whenever they add Spark files to `sys.path`,
which happens on every new worker's first task. That re-reads the JAR's
directory again: directly on Python 3.11/3.12, and on 3.13+ through the
`importlib.metadata` import that `PathFinder.invalidate_caches()` performs.
Measured `setup_spark_files` time after `import pyspark.worker`, with a
clean `pyspark.zip` built from master and the 4.0.0 spark-core JAR (median of
21 fresh processes, macOS arm64):
| Python | with JAR | without JAR |
|---|---|---|
| 3.11 | 57.7 ms | 44.0 ms |
| 3.12 | 50.3 ms | 42.3 ms |
| 3.13 | 13.5 ms | 4.9 ms |
| 3.14 | 13.2 ms | 4.8 ms |
This is paid per new Python worker process, or per task when
`spark.python.worker.reuse` is disabled. (The remaining time on 3.11/3.12 comes
from re-reading `pyspark.zip` itself and will be addressed separately.)
### Does this PR introduce _any_ user-facing change?
Yes, for a narrow case. `jarOfObject(this)` returns the jar that contains
`PythonUtils`, which is not always the spark-core jar: if an application
bundles Spark's classes into its own jar, such as a shaded uber-jar launched
with `java -cp app.jar ...`, the entry was the application jar itself, whether
or not `SPARK_HOME` was set. Python modules bundled at the root of such a jar
used to be importable through `zipimport`, and after this change they fail with
`ModuleNotFoundError`. Such [...]
For every other deployment there is no change: no Spark build since 1.4 has
put Python code in its jars, so the entry could not resolve any import for
`pyspark` or py4j.
### How was this patch tested?
- `build/sbt core/compile`, and checked with `javap` that `PythonUtils$`,
both in `core/target` and in the jar under `assembly/target/scala-2.13/jars`,
no longer references `jarOfObject`.
- Called `PythonUtils.sparkPythonPath` directly: with `SPARK_HOME` unset it
returns an empty string, and with `SPARK_HOME` set it returns exactly
`$SPARK_HOME/python/lib/pyspark.zip` and the py4j zip. On master, with the SBT
core test classpath and `SPARK_HOME` unset, it returns the spark-core jar.
- `build/sbt 'core/testOnly org.apache.spark.deploy.PythonRunnerSuite
org.apache.spark.api.python.PythonWorkerFactorySuite
org.apache.spark.api.python.PythonRDDSuite'`: 13 passed (including the new test
below), and after `build/sbt -Phive package`, `python/run-tests --testnames
'pyspark.sql.tests.test_python_worker_env,pyspark.tests.test_worker'` passed on
Python 3.12 (4 platform-gated skips). Apart from the new test, these only check
that nothing else regressed: none of them asserts [...]
- Added a test to `PythonRunnerSuite` that asserts `sparkPythonPath` has no
`.jar` entry and is exactly the two zips under `SPARK_HOME`, or empty when it
is unset. It compares the whole string, so an empty entry (which Python reads
as the current directory) also fails. Because SBT loads `PythonUtils` from the
spark-core jar, it is effective under SBT, which CI uses: locally, with
master's `PythonUtils.scala` it fails with `SPARK_HOME` both unset and set (the
extra entry is `.../core/t [...]
- End to end in a jar-packaged environment: a one-task job run via
`bin/spark-submit` (classes loaded from `assembly/target/scala-2.13/jars`, no
prepended classes) that returns the worker's `sys.path`. On master, both the
driver's and the worker's `sys.path` contain `spark-core_2.13-*.jar`. With this
change, neither has a `.jar` entry, and both still contain `pyspark.zip` and
the py4j zip.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
Closes #60071 from gaogaotiantian/remove-jar-from-pythonpath.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit 0e6bae7cc5c2801540a7ffd201d77b4f778579d7)
Signed-off-by: Tian Gao <[email protected]>
---
.../org/apache/spark/api/python/PythonUtils.scala | 21 ++++++++++-----------
.../org/apache/spark/deploy/PythonRunner.scala | 4 ++--
.../org/apache/spark/deploy/PythonRunnerSuite.scala | 19 ++++++++++++++++++-
docs/configuration.md | 8 +++++---
docs/core-migration-guide.md | 2 ++
project/SparkBuild.scala | 5 ++---
python/pyspark/sql/tests/test_python_worker_env.py | 8 ++++----
.../execution/python/PythonWorkerEnvironment.scala | 5 ++---
8 files changed, 45 insertions(+), 27 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonUtils.scala
b/core/src/main/scala/org/apache/spark/api/python/PythonUtils.scala
index 530db44184cb..cea4afaf513b 100644
--- a/core/src/main/scala/org/apache/spark/api/python/PythonUtils.scala
+++ b/core/src/main/scala/org/apache/spark/api/python/PythonUtils.scala
@@ -22,11 +22,10 @@ import java.nio.file.Paths
import java.util.{List => JList}
import scala.collection.mutable
-import scala.collection.mutable.ArrayBuffer
import scala.jdk.CollectionConverters._
import scala.sys.process.Process
-import org.apache.spark.{SparkContext, SparkEnv}
+import org.apache.spark.SparkEnv
import org.apache.spark.api.java.{JavaRDD, JavaSparkContext}
import org.apache.spark.internal.Logging
import org.apache.spark.internal.LogKeys.{PATH, PYTHON_PACKAGES,
PYTHON_VERSION}
@@ -37,16 +36,16 @@ import org.apache.spark.util.Utils
private[spark] object PythonUtils extends Logging {
val PY4J_ZIP_NAME = "py4j-0.10.9.9-src.zip"
- /** Get the PYTHONPATH for PySpark, either from SPARK_HOME, if it is set, or
from our JAR */
+ /**
+ * Get the PYTHONPATH for PySpark from SPARK_HOME, if it is set. Returns an
empty string
+ * otherwise, so callers should combine it with other paths via
`mergePythonPaths`.
+ */
def sparkPythonPath: String = {
- val pythonPath = new ArrayBuffer[String]
- for (sparkHome <- sys.env.get("SPARK_HOME")) {
- pythonPath += Seq(sparkHome, "python", "lib",
"pyspark.zip").mkString(File.separator)
- pythonPath +=
- Seq(sparkHome, "python", "lib", PY4J_ZIP_NAME).mkString(File.separator)
- }
- pythonPath ++= SparkContext.jarOfObject(this)
- pythonPath.mkString(File.pathSeparator)
+ sys.env.get("SPARK_HOME").map { sparkHome =>
+ Seq("pyspark.zip", PY4J_ZIP_NAME)
+ .map(name => Seq(sparkHome, "python", "lib",
name).mkString(File.separator))
+ .mkString(File.pathSeparator)
+ }.getOrElse("")
}
/** Merge PYTHONPATHS with the appropriate separator. Ignores blank strings.
*/
diff --git a/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
b/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
index 87cef4d242ab..b6665a7f574f 100644
--- a/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
@@ -73,8 +73,8 @@ object PythonRunner {
thread.join()
}
- // Build up a PYTHONPATH that includes the Spark assembly (where this
class is), the
- // python directories in SPARK_HOME (if set), and any files in the pyFiles
argument
+ // Build up a PYTHONPATH that includes any files in the pyFiles argument,
the python
+ // directories in SPARK_HOME (if set), and the PYTHONPATH inherited from
the environment
val pathElements = new ArrayBuffer[String]
pathElements ++= formattedPyFiles
pathElements += PythonUtils.sparkPythonPath
diff --git
a/core/src/test/scala/org/apache/spark/deploy/PythonRunnerSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/PythonRunnerSuite.scala
index 2cce3d306e60..45dccfa40169 100644
--- a/core/src/test/scala/org/apache/spark/deploy/PythonRunnerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/PythonRunnerSuite.scala
@@ -17,8 +17,10 @@
package org.apache.spark.deploy
+import java.io.File
+
import org.apache.spark.{SparkFunSuite, SparkThrowable}
-import org.apache.spark.api.python.PythonErrorUtils
+import org.apache.spark.api.python.{PythonErrorUtils, PythonUtils}
import org.apache.spark.util.Utils
class PythonRunnerSuite extends SparkFunSuite {
@@ -66,6 +68,21 @@ class PythonRunnerSuite extends SparkFunSuite {
intercept[IllegalArgumentException] {
PythonRunner.formatPaths("foo.py,hdfs:/some.py") }
}
+ // Test the paths Spark itself adds to the PYTHONPATH
+ test("SPARK-60855: sparkPythonPath should only contain the zips under
SPARK_HOME") {
+ val sparkPythonPath = PythonUtils.sparkPythonPath
+ // Spark's jars contain no Python code, so the jar containing PythonUtils
must not be added.
+ // SBT loads PythonUtils from the spark-core jar, which makes this catch
it being added back.
+
assert(!sparkPythonPath.split(File.pathSeparator).exists(_.endsWith(".jar")),
sparkPythonPath)
+ // Compare the whole string, so that an empty entry, which Python reads as
the current
+ // directory, also fails. It is empty when SPARK_HOME is unset.
+ val expected = sys.env.get("SPARK_HOME").toSeq.flatMap { sparkHome =>
+ Seq("pyspark.zip", PythonUtils.PY4J_ZIP_NAME)
+ .map(name => Seq(sparkHome, "python", "lib",
name).mkString(File.separator))
+ }
+ assert(sparkPythonPath === expected.mkString(File.pathSeparator))
+ }
+
test("SPARK-54052: PythonErrorUtils should have corresponding methods in
SparkThrowable") {
// Find default methods in SparkThrowable
val defaultMethods = classOf[SparkThrowable]
diff --git a/docs/configuration.md b/docs/configuration.md
index 97f11a334d8b..b03ab4615c37 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -865,9 +865,11 @@ Apart from these, the following properties are also
available, and may be useful
<code>PYTHON_UDF_BATCH_SIZE</code>, a session may still set it and Spark's
value wins.
<code>PYTHONPATH</code> is the one name that is neither reserved nor
simply overridden: Spark
merges the session's value into the path it computes for the worker, so a
session adds to the
- worker's import path. The relative order is not guaranteed -- Spark
contributes no entries of
- its own when <code>SPARK_HOME</code> is unset and its classes did not come
from a jar, which can
- leave the session's path first -- so do not rely on a session entry being
shadowed by Spark's.
+ worker's import path. The relative order is not guaranteed -- Spark puts
no entries of its own
+ ahead of the session's when <code>SPARK_HOME</code> is unset where the
worker runs, as on YARN
+ executors, which receive Spark's <code>pyspark.zip</code> through the
inherited
+ <code>PYTHONPATH</code> after the session's entries -- so do not rely on a
session entry being
+ shadowed by Spark's.
<br /><br />
Each variable is a separate configuration, so setting several is not
atomic: a client that sets
a batch may have some applied and then one rejected, leaving the earlier
ones in place. Read the
diff --git a/docs/core-migration-guide.md b/docs/core-migration-guide.md
index cda1a1f03638..aebedc7345d0 100644
--- a/docs/core-migration-guide.md
+++ b/docs/core-migration-guide.md
@@ -32,6 +32,8 @@ license: |
- Since Spark 4.4, Spark accurately records skewed shuffle block sizes in
`HighlyCompressedMapStatus`, so that adaptive query execution can still detect
skewed partitions in shuffles with more than 2000 partitions. The default of
`spark.shuffle.accurateBlockSkewedFactor` changed from `-1.0` to `5.0`,
matching `spark.sql.adaptive.skewJoin.skewedPartitionFactor`. This can change
query plans, since such skewed partitions are now split by skew join
optimization, and it costs driver memory: u [...]
+- Since Spark 4.4, Spark no longer adds the jar that contains its own classes
to the `PYTHONPATH` of the Python processes it launches, such as Python workers
and the driver process started by `spark-submit`. Spark's jars have contained
no Python code since Spark 1.4, so this only matters to an application that
bundles Spark's classes into its own jar, such as a shaded uber-jar launched
with `java -cp app.jar`, and also bundles Python modules in that jar: those
modules are no longer impor [...]
+
## Upgrading from Core 4.2 to 4.3
- Since Spark 4.3, when submitting to a standalone cluster in cluster mode
through the RPC submission gateway, which is what `spark-submit` uses unless
`spark.master.rest.enabled` is set to `true`, Spark forwards to the driver only
the environment variables whose name starts with `SPARK_`, excluding
`SPARK_ENV_LOADED`, `SPARK_HOME`, `SPARK_CONF_DIR`, `SPARK_LOCAL_IP`, and
`SPARK_LOCAL_HOSTNAME`, matching the REST submission gateway. Previously the
full environment of the submitting proce [...]
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index abe026be1b53..38b8be33d052 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1593,9 +1593,8 @@ object PySparkAssembly {
import java.util.zip.{ZipOutputStream, ZipEntry}
lazy val settings = Seq(
- // Use a resource generator to copy all .py files from python/pyspark into
a managed directory
- // to be included in the assembly. We can't just add "python/" to the
assembly's resource dir
- // list since that will copy unneeded / unwanted files.
+ // Use a resource generator to zip python/pyspark into
python/lib/pyspark.zip. It returns no
+ // files, so nothing from python/ is added to the assembly jar.
(Compile / resourceGenerators) += Def.macroValueI((Compile /
resourceManaged) map { outDir: File =>
val src = new File(BuildCommons.sparkHome, "python/pyspark")
val zipFile = new File(BuildCommons.sparkHome , "python/lib/pyspark.zip")
diff --git a/python/pyspark/sql/tests/test_python_worker_env.py
b/python/pyspark/sql/tests/test_python_worker_env.py
index 43ad3642da6b..ea03ce27ee6a 100644
--- a/python/pyspark/sql/tests/test_python_worker_env.py
+++ b/python/pyspark/sql/tests/test_python_worker_env.py
@@ -142,10 +142,10 @@ class PythonWorkerEnvMixin:
# PYTHONPATH is neither reserved nor overridden: PythonWorkerFactory
folds the session's
# value into the path it computes, so the entry reaches the worker
rather than being
# dropped. That is the whole guarantee. The relative order is
deliberately not asserted:
- # `sparkPythonPath` is empty when SPARK_HOME is unset and Spark's
classes did not come from
- # a jar, and `mergePythonPaths` drops empty entries, so the session's
path can legitimately
- # come first. Leaving the name settable rests on the session already
choosing the code its
- # own worker runs, not on Spark's entries winning.
+ # `sparkPythonPath` is empty when SPARK_HOME is unset, and
`mergePythonPaths` drops empty
+ # entries, so the session's path can legitimately come first. Leaving
the name settable
+ # rests on the session already choosing the code its own worker runs,
not on Spark's
+ # entries winning.
marker = "/tmp/spark-58752-extra-modules"
self._set_env("PYTHONPATH", marker)
try:
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonWorkerEnvironment.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonWorkerEnvironment.scala
index adfb6cd00033..c5008e41561c 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonWorkerEnvironment.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonWorkerEnvironment.scala
@@ -91,9 +91,8 @@ private[sql] object PythonWorkerEnvironment {
* `PythonWorkerFactory` folds the session's value into the path it computes
instead of discarding
* it, so a session adds to the worker's import path. It is left settable
because a session
* already chooses the code its own worker runs, not because Spark's entries
are guaranteed to
- * come first -- `PythonUtils.sparkPythonPath` is empty when `SPARK_HOME` is
unset and Spark's
- * classes did not come from a jar, and `mergePythonPaths` drops empty
entries, which can leave
- * the session's path first.
+ * come first -- `PythonUtils.sparkPythonPath` is empty when `SPARK_HOME` is
unset, and
+ * `mergePythonPaths` drops empty entries, which can leave the session's
path first.
*
* These are the ones write order does not protect, because Spark sets them
only when a condition
* holds and a session's value survives when it does not. Two of them would
desynchronize the JVM
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]