This is an automated email from the ASF dual-hosted git repository.

gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 3005717da8c6 [SPARK-60855][CORE] Remove the Spark JAR from the Python 
worker PYTHONPATH
3005717da8c6 is described below

commit 3005717da8c67fdff7b193e6644a57b6d46a2152
Author: Tian Gao <[email protected]>
AuthorDate: Thu Nov 5 10:42:44 2026 -0800

    [SPARK-60855][CORE] Remove the Spark JAR from the Python worker PYTHONPATH
    
    ### What changes were proposed in this pull request?
    
    Remove `SparkContext.jarOfObject(this)` from `PythonUtils.sparkPythonPath`, 
so the JAR containing Spark's classes (normally the spark-core JAR) is no 
longer put on the `PYTHONPATH` of Python processes Spark launches (Python 
workers via `PythonWorkerFactory`, `PythonRunner`, 
`PythonUDFWorkerSpecification`), nor on the `sys.path` of the embedded 
interpreter `InProcessPythonRuntime` sets up. `sparkPythonPath` now contains 
only `$SPARK_HOME/python/lib/pyspark.zip` and the py4j zip, or not [...]
    
    The comments and docs that described the JAR entry are updated too: the 
`sparkPythonPath` scaladoc, `PythonRunner`, `PythonWorkerEnvironment`, 
`test_python_worker_env.py`, `docs/configuration.md`, and the stale 
`PySparkAssembly` comment in `project/SparkBuild.scala`. A core migration guide 
entry is added for the one case that can notice the change (see below), and a 
regression test in `PythonRunnerSuite` guards against the jar entry coming back.
    
    ### Why are the changes needed?
    
    **The JAR has not contained any Python code since Spark 1.4.** Here is how 
the entry got there and why it is dead:
    
    1. **SPARK-1004 (ff5be9a41e5, April 2014), PySpark on YARN.** To run 
PySpark on YARN executors, which have no `SPARK_HOME`, `core/pom.xml` (and 
`mllib/pom.xml`) copied `pyspark/*.py` and the unzipped py4j sources into the 
JAR as resources. Python can import from a JAR because a JAR is a zip file 
(`zipimport`).
    2. **SPARK-1549 (951a5d93986, May 2014), Python support in spark-submit.** 
This introduced `PythonUtils.sparkPythonPath`, documented as "Get the 
PYTHONPATH for PySpark, either from SPARK_HOME, if it is set, or from our JAR". 
The commit message says: "we assume we can always find the Python files either 
from the Spark assembly JAR ... or from SPARK_HOME". `jarOfObject(this)` (the 
JAR containing `PythonUtils`) was added to the path for that reason. SPARK-3008 
(2014, "zipimport not able  [...]
    3. **SPARK-6869 (ebff7327af5, May 2015).** YARN started shipping 
`pyspark.zip` and the py4j zip to containers as separate archives and putting 
them on the executors' `PYTHONPATH` (today's `Client.findPySparkArchives` / 
`PYSPARK_ARCHIVES_PATH`).
    4. **SPARK-7485 (82e890fb19d, May 2015), "Remove pyspark files from 
assembly".** This removed the Maven resource rules and the py4j unzip step, and 
made the SBT `PySparkAssembly` generator build only `python/lib/pyspark.zip` 
(it returns `Seq.empty`, so nothing goes into the JAR). The `sparkPythonPath` 
line was not updated, and the follow-up 98478752662 only changed the 
`SPARK_HOME` entry to `pyspark.zip`.
    
    Since then, no Spark build has packaged Python into a JAR. On current 
master, `assembly/pom.xml` and `project/SparkBuild.scala` build `pyspark.zip` 
only, and no `.py` file exists under any `src/main/resources`. The published 
`spark-core_2.13-4.0.0.jar` has 5,261 entries and no `.py`/`.pyc` files. Every 
deployment gets PySpark some other way:
    
    | Deployment | Where `pyspark` is imported from |
    |---|---|
    | Local / standalone / Kubernetes (distribution or image) | 
`$SPARK_HOME/python/lib/pyspark.zip` (this same method) |
    | YARN | `pyspark.zip` and py4j shipped by `Client` and set in the 
containers' `PYTHONPATH` |
    | pip-installed PySpark | site-packages |
    
    So with any Spark-built JAR, the entry can never resolve an import (the one 
exception, an application uber-jar with Spark shaded in, is covered below). It 
only adds work:
    
    - Python creates a `zipimporter` for the JAR and reads its central 
directory (one entry per class file).
    - `PYTHONPATH` entries come before the standard library on `sys.path`, so 
every top-level import that misses `pyspark.zip` and py4j also probes the JAR.
    - Python workers call `importlib.invalidate_caches()` in 
`worker_util.setup_spark_files` whenever they add Spark files to `sys.path`, 
which happens on every new worker's first task. That re-reads the JAR's 
directory again: directly on Python 3.11/3.12, and on 3.13+ through the 
`importlib.metadata` import that `PathFinder.invalidate_caches()` performs.
    
    Measured `setup_spark_files` time after `import pyspark.worker`, with a 
clean `pyspark.zip` built from master and the 4.0.0 spark-core JAR (median of 
21 fresh processes, macOS arm64):
    
    | Python | with JAR | without JAR |
    |---|---|---|
    | 3.11 | 57.7 ms | 44.0 ms |
    | 3.12 | 50.3 ms | 42.3 ms |
    | 3.13 | 13.5 ms | 4.9 ms |
    | 3.14 | 13.2 ms | 4.8 ms |
    
    This is paid per new Python worker process, or per task when 
`spark.python.worker.reuse` is disabled. (The remaining time on 3.11/3.12 comes 
from re-reading `pyspark.zip` itself and will be addressed separately.)
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, for a narrow case. `jarOfObject(this)` returns the jar that contains 
`PythonUtils`, which is not always the spark-core jar: if an application 
bundles Spark's classes into its own jar, such as a shaded uber-jar launched 
with `java -cp app.jar ...`, the entry was the application jar itself, whether 
or not `SPARK_HOME` was set. Python modules bundled at the root of such a jar 
used to be importable through `zipimport`, and after this change they fail with 
`ModuleNotFoundError`. Such  [...]
    
    For every other deployment there is no change: no Spark build since 1.4 has 
put Python code in its jars, so the entry could not resolve any import for 
`pyspark` or py4j.
    
    ### How was this patch tested?
    
    - `build/sbt core/compile`, and checked with `javap` that `PythonUtils$`, 
both in `core/target` and in the jar under `assembly/target/scala-2.13/jars`, 
no longer references `jarOfObject`.
    - Called `PythonUtils.sparkPythonPath` directly: with `SPARK_HOME` unset it 
returns an empty string, and with `SPARK_HOME` set it returns exactly 
`$SPARK_HOME/python/lib/pyspark.zip` and the py4j zip. On master, with the SBT 
core test classpath and `SPARK_HOME` unset, it returns the spark-core jar.
    - `build/sbt 'core/testOnly org.apache.spark.deploy.PythonRunnerSuite 
org.apache.spark.api.python.PythonWorkerFactorySuite 
org.apache.spark.api.python.PythonRDDSuite'`: 13 passed (including the new test 
below), and after `build/sbt -Phive package`, `python/run-tests --testnames 
'pyspark.sql.tests.test_python_worker_env,pyspark.tests.test_worker'` passed on 
Python 3.12 (4 platform-gated skips). Apart from the new test, these only check 
that nothing else regressed: none of them asserts  [...]
    - Added a test to `PythonRunnerSuite` that asserts `sparkPythonPath` has no 
`.jar` entry and is exactly the two zips under `SPARK_HOME`, or empty when it 
is unset. It compares the whole string, so an empty entry (which Python reads 
as the current directory) also fails. Because SBT loads `PythonUtils` from the 
spark-core jar, it is effective under SBT, which CI uses: locally, with 
master's `PythonUtils.scala` it fails with `SPARK_HOME` both unset and set (the 
extra entry is `.../core/t [...]
    - End to end in a jar-packaged environment: a one-task job run via 
`bin/spark-submit` (classes loaded from `assembly/target/scala-2.13/jars`, no 
prepended classes) that returns the worker's `sys.path`. On master, both the 
driver's and the worker's `sys.path` contain `spark-core_2.13-*.jar`. With this 
change, neither has a `.jar` entry, and both still contain `pyspark.zip` and 
the py4j zip.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Isaac
    
    This pull request and its description were written by Isaac.
    
    Closes #60071 from gaogaotiantian/remove-jar-from-pythonpath.
    
    Authored-by: Tian Gao <[email protected]>
    Signed-off-by: Tian Gao <[email protected]>
    (cherry picked from commit 0e6bae7cc5c2801540a7ffd201d77b4f778579d7)
    Signed-off-by: Tian Gao <[email protected]>
---
 .../org/apache/spark/api/python/PythonUtils.scala   | 21 ++++++++++-----------
 .../org/apache/spark/deploy/PythonRunner.scala      |  4 ++--
 .../org/apache/spark/deploy/PythonRunnerSuite.scala | 19 ++++++++++++++++++-
 docs/configuration.md                               |  8 +++++---
 docs/core-migration-guide.md                        |  2 ++
 project/SparkBuild.scala                            |  5 ++---
 python/pyspark/sql/tests/test_python_worker_env.py  |  8 ++++----
 .../execution/python/PythonWorkerEnvironment.scala  |  5 ++---
 8 files changed, 45 insertions(+), 27 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonUtils.scala 
b/core/src/main/scala/org/apache/spark/api/python/PythonUtils.scala
index 530db44184cb..cea4afaf513b 100644
--- a/core/src/main/scala/org/apache/spark/api/python/PythonUtils.scala
+++ b/core/src/main/scala/org/apache/spark/api/python/PythonUtils.scala
@@ -22,11 +22,10 @@ import java.nio.file.Paths
 import java.util.{List => JList}
 
 import scala.collection.mutable
-import scala.collection.mutable.ArrayBuffer
 import scala.jdk.CollectionConverters._
 import scala.sys.process.Process
 
-import org.apache.spark.{SparkContext, SparkEnv}
+import org.apache.spark.SparkEnv
 import org.apache.spark.api.java.{JavaRDD, JavaSparkContext}
 import org.apache.spark.internal.Logging
 import org.apache.spark.internal.LogKeys.{PATH, PYTHON_PACKAGES, 
PYTHON_VERSION}
@@ -37,16 +36,16 @@ import org.apache.spark.util.Utils
 private[spark] object PythonUtils extends Logging {
   val PY4J_ZIP_NAME = "py4j-0.10.9.9-src.zip"
 
-  /** Get the PYTHONPATH for PySpark, either from SPARK_HOME, if it is set, or 
from our JAR */
+  /**
+   * Get the PYTHONPATH for PySpark from SPARK_HOME, if it is set. Returns an 
empty string
+   * otherwise, so callers should combine it with other paths via 
`mergePythonPaths`.
+   */
   def sparkPythonPath: String = {
-    val pythonPath = new ArrayBuffer[String]
-    for (sparkHome <- sys.env.get("SPARK_HOME")) {
-      pythonPath += Seq(sparkHome, "python", "lib", 
"pyspark.zip").mkString(File.separator)
-      pythonPath +=
-        Seq(sparkHome, "python", "lib", PY4J_ZIP_NAME).mkString(File.separator)
-    }
-    pythonPath ++= SparkContext.jarOfObject(this)
-    pythonPath.mkString(File.pathSeparator)
+    sys.env.get("SPARK_HOME").map { sparkHome =>
+      Seq("pyspark.zip", PY4J_ZIP_NAME)
+        .map(name => Seq(sparkHome, "python", "lib", 
name).mkString(File.separator))
+        .mkString(File.pathSeparator)
+    }.getOrElse("")
   }
 
   /** Merge PYTHONPATHS with the appropriate separator. Ignores blank strings. 
*/
diff --git a/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala 
b/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
index 87cef4d242ab..b6665a7f574f 100644
--- a/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala
@@ -73,8 +73,8 @@ object PythonRunner {
       thread.join()
     }
 
-    // Build up a PYTHONPATH that includes the Spark assembly (where this 
class is), the
-    // python directories in SPARK_HOME (if set), and any files in the pyFiles 
argument
+    // Build up a PYTHONPATH that includes any files in the pyFiles argument, 
the python
+    // directories in SPARK_HOME (if set), and the PYTHONPATH inherited from 
the environment
     val pathElements = new ArrayBuffer[String]
     pathElements ++= formattedPyFiles
     pathElements += PythonUtils.sparkPythonPath
diff --git 
a/core/src/test/scala/org/apache/spark/deploy/PythonRunnerSuite.scala 
b/core/src/test/scala/org/apache/spark/deploy/PythonRunnerSuite.scala
index 2cce3d306e60..45dccfa40169 100644
--- a/core/src/test/scala/org/apache/spark/deploy/PythonRunnerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/PythonRunnerSuite.scala
@@ -17,8 +17,10 @@
 
 package org.apache.spark.deploy
 
+import java.io.File
+
 import org.apache.spark.{SparkFunSuite, SparkThrowable}
-import org.apache.spark.api.python.PythonErrorUtils
+import org.apache.spark.api.python.{PythonErrorUtils, PythonUtils}
 import org.apache.spark.util.Utils
 
 class PythonRunnerSuite extends SparkFunSuite {
@@ -66,6 +68,21 @@ class PythonRunnerSuite extends SparkFunSuite {
     intercept[IllegalArgumentException] { 
PythonRunner.formatPaths("foo.py,hdfs:/some.py") }
   }
 
+  // Test the paths Spark itself adds to the PYTHONPATH
+  test("SPARK-60855: sparkPythonPath should only contain the zips under 
SPARK_HOME") {
+    val sparkPythonPath = PythonUtils.sparkPythonPath
+    // Spark's jars contain no Python code, so the jar containing PythonUtils 
must not be added.
+    // SBT loads PythonUtils from the spark-core jar, which makes this catch 
it being added back.
+    
assert(!sparkPythonPath.split(File.pathSeparator).exists(_.endsWith(".jar")), 
sparkPythonPath)
+    // Compare the whole string, so that an empty entry, which Python reads as 
the current
+    // directory, also fails. It is empty when SPARK_HOME is unset.
+    val expected = sys.env.get("SPARK_HOME").toSeq.flatMap { sparkHome =>
+      Seq("pyspark.zip", PythonUtils.PY4J_ZIP_NAME)
+        .map(name => Seq(sparkHome, "python", "lib", 
name).mkString(File.separator))
+    }
+    assert(sparkPythonPath === expected.mkString(File.pathSeparator))
+  }
+
   test("SPARK-54052: PythonErrorUtils should have corresponding methods in 
SparkThrowable") {
     // Find default methods in SparkThrowable
     val defaultMethods = classOf[SparkThrowable]
diff --git a/docs/configuration.md b/docs/configuration.md
index 97f11a334d8b..b03ab4615c37 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -865,9 +865,11 @@ Apart from these, the following properties are also 
available, and may be useful
     <code>PYTHON_UDF_BATCH_SIZE</code>, a session may still set it and Spark's 
value wins.
     <code>PYTHONPATH</code> is the one name that is neither reserved nor 
simply overridden: Spark
     merges the session's value into the path it computes for the worker, so a 
session adds to the
-    worker's import path. The relative order is not guaranteed -- Spark 
contributes no entries of
-    its own when <code>SPARK_HOME</code> is unset and its classes did not come 
from a jar, which can
-    leave the session's path first -- so do not rely on a session entry being 
shadowed by Spark's.
+    worker's import path. The relative order is not guaranteed -- Spark puts 
no entries of its own
+    ahead of the session's when <code>SPARK_HOME</code> is unset where the 
worker runs, as on YARN
+    executors, which receive Spark's <code>pyspark.zip</code> through the 
inherited
+    <code>PYTHONPATH</code> after the session's entries -- so do not rely on a 
session entry being
+    shadowed by Spark's.
     <br /><br />
     Each variable is a separate configuration, so setting several is not 
atomic: a client that sets
     a batch may have some applied and then one rejected, leaving the earlier 
ones in place. Read the
diff --git a/docs/core-migration-guide.md b/docs/core-migration-guide.md
index cda1a1f03638..aebedc7345d0 100644
--- a/docs/core-migration-guide.md
+++ b/docs/core-migration-guide.md
@@ -32,6 +32,8 @@ license: |
 
 - Since Spark 4.4, Spark accurately records skewed shuffle block sizes in 
`HighlyCompressedMapStatus`, so that adaptive query execution can still detect 
skewed partitions in shuffles with more than 2000 partitions. The default of 
`spark.shuffle.accurateBlockSkewedFactor` changed from `-1.0` to `5.0`, 
matching `spark.sql.adaptive.skewJoin.skewedPartitionFactor`. This can change 
query plans, since such skewed partitions are now split by skew join 
optimization, and it costs driver memory: u [...]
 
+- Since Spark 4.4, Spark no longer adds the jar that contains its own classes 
to the `PYTHONPATH` of the Python processes it launches, such as Python workers 
and the driver process started by `spark-submit`. Spark's jars have contained 
no Python code since Spark 1.4, so this only matters to an application that 
bundles Spark's classes into its own jar, such as a shaded uber-jar launched 
with `java -cp app.jar`, and also bundles Python modules in that jar: those 
modules are no longer impor [...]
+
 ## Upgrading from Core 4.2 to 4.3
 
 - Since Spark 4.3, when submitting to a standalone cluster in cluster mode 
through the RPC submission gateway, which is what `spark-submit` uses unless 
`spark.master.rest.enabled` is set to `true`, Spark forwards to the driver only 
the environment variables whose name starts with `SPARK_`, excluding 
`SPARK_ENV_LOADED`, `SPARK_HOME`, `SPARK_CONF_DIR`, `SPARK_LOCAL_IP`, and 
`SPARK_LOCAL_HOSTNAME`, matching the REST submission gateway. Previously the 
full environment of the submitting proce [...]
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index abe026be1b53..38b8be33d052 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1593,9 +1593,8 @@ object PySparkAssembly {
   import java.util.zip.{ZipOutputStream, ZipEntry}
 
   lazy val settings = Seq(
-    // Use a resource generator to copy all .py files from python/pyspark into 
a managed directory
-    // to be included in the assembly. We can't just add "python/" to the 
assembly's resource dir
-    // list since that will copy unneeded / unwanted files.
+    // Use a resource generator to zip python/pyspark into 
python/lib/pyspark.zip. It returns no
+    // files, so nothing from python/ is added to the assembly jar.
     (Compile / resourceGenerators) += Def.macroValueI((Compile / 
resourceManaged) map { outDir: File =>
       val src = new File(BuildCommons.sparkHome, "python/pyspark")
       val zipFile = new File(BuildCommons.sparkHome , "python/lib/pyspark.zip")
diff --git a/python/pyspark/sql/tests/test_python_worker_env.py 
b/python/pyspark/sql/tests/test_python_worker_env.py
index 43ad3642da6b..ea03ce27ee6a 100644
--- a/python/pyspark/sql/tests/test_python_worker_env.py
+++ b/python/pyspark/sql/tests/test_python_worker_env.py
@@ -142,10 +142,10 @@ class PythonWorkerEnvMixin:
         # PYTHONPATH is neither reserved nor overridden: PythonWorkerFactory 
folds the session's
         # value into the path it computes, so the entry reaches the worker 
rather than being
         # dropped. That is the whole guarantee. The relative order is 
deliberately not asserted:
-        # `sparkPythonPath` is empty when SPARK_HOME is unset and Spark's 
classes did not come from
-        # a jar, and `mergePythonPaths` drops empty entries, so the session's 
path can legitimately
-        # come first. Leaving the name settable rests on the session already 
choosing the code its
-        # own worker runs, not on Spark's entries winning.
+        # `sparkPythonPath` is empty when SPARK_HOME is unset, and 
`mergePythonPaths` drops empty
+        # entries, so the session's path can legitimately come first. Leaving 
the name settable
+        # rests on the session already choosing the code its own worker runs, 
not on Spark's
+        # entries winning.
         marker = "/tmp/spark-58752-extra-modules"
         self._set_env("PYTHONPATH", marker)
         try:
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonWorkerEnvironment.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonWorkerEnvironment.scala
index adfb6cd00033..c5008e41561c 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonWorkerEnvironment.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonWorkerEnvironment.scala
@@ -91,9 +91,8 @@ private[sql] object PythonWorkerEnvironment {
    * `PythonWorkerFactory` folds the session's value into the path it computes 
instead of discarding
    * it, so a session adds to the worker's import path. It is left settable 
because a session
    * already chooses the code its own worker runs, not because Spark's entries 
are guaranteed to
-   * come first -- `PythonUtils.sparkPythonPath` is empty when `SPARK_HOME` is 
unset and Spark's
-   * classes did not come from a jar, and `mergePythonPaths` drops empty 
entries, which can leave
-   * the session's path first.
+   * come first -- `PythonUtils.sparkPythonPath` is empty when `SPARK_HOME` is 
unset, and
+   * `mergePythonPaths` drops empty entries, which can leave the session's 
path first.
    *
    * These are the ones write order does not protect, because Spark sets them 
only when a condition
    * holds and a session's value survives when it does not. Two of them would 
desynchronize the JVM


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to