This is an automated email from the ASF dual-hosted git repository.

cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 097a83a05e1c [SPARK-57271][PYTHON] Propagate traceback locals to 
Python planner runner
097a83a05e1c is described below

commit 097a83a05e1c7c5bfe4482317ec42af3976572ed
Author: Linhong Liu <[email protected]>
AuthorDate: Wed Jul 29 20:52:09 2026 +0800

    [SPARK-57271][PYTHON] Propagate traceback locals to Python planner runner
    
    ### What changes were proposed in this pull request?
    
    This PR propagates `SPARK_TRACEBACK_WITH_LOCALS` from `PythonPlannerRunner` 
when `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled` is enabled.
    
    It also adds a UDTF `analyze` regression test that raises from the 
planner-side Python worker and verifies the surfaced traceback includes the 
local variable.
    
    ### Why are the changes needed?
    
    `PythonPlannerRunner` already reads 
`spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled`, but it did not 
add `SPARK_TRACEBACK_WITH_LOCALS` to the Python worker environment. As a 
result, planner-driven Python paths such as UDTF `analyze` did not honor the 
traceback-locals config, unlike regular Python UDF execution.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. When `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled` is 
enabled, Python exceptions raised through planner-driven Python paths can now 
include local variables in their tracebacks.
    
    ### How was this patch tested?
    
    Passed:
    
    ```
    python3 python/run-tests.py --testnames "pyspark.sql.tests.test_udtf 
UDTFTests.test_udtf_analyze_traceback_with_locals"
    python3 python/run-tests.py --testnames "pyspark.sql.tests.test_udf 
UDFTests.test_udf_traceback_with_locals"
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-5)
    
    Closes #57509 from 
linhongliu-db/task/oss-spark-spark-57271-pythonplannerrunner-which-doesn-t-extend-basepythonrunner-f822ad87ff/implementation.
    
    Authored-by: Linhong Liu <[email protected]>
    Signed-off-by: Wenchen Fan <[email protected]>
    (cherry picked from commit 87dbc2421b703d1df08ac39928151d96d49369ff)
    Signed-off-by: Wenchen Fan <[email protected]>
---
 python/pyspark/sql/tests/test_udtf.py                 | 19 +++++++++++++++++++
 .../sql/execution/python/PythonPlannerRunner.scala    |  3 +++
 2 files changed, 22 insertions(+)

diff --git a/python/pyspark/sql/tests/test_udtf.py 
b/python/pyspark/sql/tests/test_udtf.py
index a272cee4c7b5..2b30e41b30db 100644
--- a/python/pyspark/sql/tests/test_udtf.py
+++ b/python/pyspark/sql/tests/test_udtf.py
@@ -1999,6 +1999,25 @@ class BaseUDTFTestsMixin:
         with self.assertRaisesRegex(AnalysisException, "Failed to analyze."):
             func().collect()
 
+    def test_udtf_analyze_traceback_with_locals(self):
+        with 
self.sql_conf({"spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled": 
True}):
+
+            class TestUDTF:
+                @staticmethod
+                def analyze() -> AnalyzeResult:
+                    local_marker = 1
+                    if local_marker:
+                        raise ValueError("boom")
+                    return AnalyzeResult(StructType().add("x", StringType()))
+
+                def eval(self):
+                    yield ("x",)
+
+            func = udtf(TestUDTF)
+
+            with self.assertRaisesRegex(AnalysisException, "local_marker = 1"):
+                func().collect()
+
     def test_udtf_with_analyze_null_literal(self):
         class TestUDTF:
             @staticmethod
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala
index 91f0b1d1203a..a30567b7b7c6 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala
@@ -91,6 +91,9 @@ abstract class PythonPlannerRunner[T](func: PythonFunction) 
extends Logging {
     if (simplifiedTraceback) {
       envVars.put("SPARK_SIMPLIFIED_TRACEBACK", "1")
     }
+    if (tracebackWithLocals) {
+      envVars.put("SPARK_TRACEBACK_WITH_LOCALS", "1")
+    }
     workerMemoryMb.foreach { memoryMb =>
       envVars.put("PYSPARK_PLANNER_MEMORY_MB", memoryMb.toString)
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to