This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 87dbc2421b70 [SPARK-57271][PYTHON] Propagate traceback locals to
Python planner runner
87dbc2421b70 is described below
commit 87dbc2421b703d1df08ac39928151d96d49369ff
Author: Linhong Liu <[email protected]>
AuthorDate: Wed Jul 29 20:52:09 2026 +0800
[SPARK-57271][PYTHON] Propagate traceback locals to Python planner runner
### What changes were proposed in this pull request?
This PR propagates `SPARK_TRACEBACK_WITH_LOCALS` from `PythonPlannerRunner`
when `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled` is enabled.
It also adds a UDTF `analyze` regression test that raises from the
planner-side Python worker and verifies the surfaced traceback includes the
local variable.
### Why are the changes needed?
`PythonPlannerRunner` already reads
`spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled`, but it did not
add `SPARK_TRACEBACK_WITH_LOCALS` to the Python worker environment. As a
result, planner-driven Python paths such as UDTF `analyze` did not honor the
traceback-locals config, unlike regular Python UDF execution.
### Does this PR introduce _any_ user-facing change?
Yes. When `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled` is
enabled, Python exceptions raised through planner-driven Python paths can now
include local variables in their tracebacks.
### How was this patch tested?
Passed:
```
python3 python/run-tests.py --testnames "pyspark.sql.tests.test_udtf
UDTFTests.test_udtf_analyze_traceback_with_locals"
python3 python/run-tests.py --testnames "pyspark.sql.tests.test_udf
UDFTests.test_udf_traceback_with_locals"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
Closes #57509 from
linhongliu-db/task/oss-spark-spark-57271-pythonplannerrunner-which-doesn-t-extend-basepythonrunner-f822ad87ff/implementation.
Authored-by: Linhong Liu <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
python/pyspark/sql/tests/test_udtf.py | 19 +++++++++++++++++++
.../sql/execution/python/PythonPlannerRunner.scala | 3 +++
2 files changed, 22 insertions(+)
diff --git a/python/pyspark/sql/tests/test_udtf.py
b/python/pyspark/sql/tests/test_udtf.py
index a272cee4c7b5..2b30e41b30db 100644
--- a/python/pyspark/sql/tests/test_udtf.py
+++ b/python/pyspark/sql/tests/test_udtf.py
@@ -1999,6 +1999,25 @@ class BaseUDTFTestsMixin:
with self.assertRaisesRegex(AnalysisException, "Failed to analyze."):
func().collect()
+ def test_udtf_analyze_traceback_with_locals(self):
+ with
self.sql_conf({"spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled":
True}):
+
+ class TestUDTF:
+ @staticmethod
+ def analyze() -> AnalyzeResult:
+ local_marker = 1
+ if local_marker:
+ raise ValueError("boom")
+ return AnalyzeResult(StructType().add("x", StringType()))
+
+ def eval(self):
+ yield ("x",)
+
+ func = udtf(TestUDTF)
+
+ with self.assertRaisesRegex(AnalysisException, "local_marker = 1"):
+ func().collect()
+
def test_udtf_with_analyze_null_literal(self):
class TestUDTF:
@staticmethod
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala
index 91f0b1d1203a..a30567b7b7c6 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala
@@ -91,6 +91,9 @@ abstract class PythonPlannerRunner[T](func: PythonFunction)
extends Logging {
if (simplifiedTraceback) {
envVars.put("SPARK_SIMPLIFIED_TRACEBACK", "1")
}
+ if (tracebackWithLocals) {
+ envVars.put("SPARK_TRACEBACK_WITH_LOCALS", "1")
+ }
workerMemoryMb.foreach { memoryMb =>
envVars.put("PYSPARK_PLANNER_MEMORY_MB", memoryMb.toString)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]