This is an automated email from the ASF dual-hosted git repository.
ueshin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 28a756222800 [SPARK-51608][PYTHON] Log exception on Python runner
termination
28a756222800 is described below
commit 28a756222800810868994c1cd6de09a9bf3d6584
Author: dmitry.sorokin <[email protected]>
AuthorDate: Fri Mar 28 13:13:20 2025 -0700
[SPARK-51608][PYTHON] Log exception on Python runner termination
### What changes were proposed in this pull request?
There are situations, when python worker may throw exceptions that are not
covered by any other kind of logging (for example custom
implementation/unexpected throw from existing code)
In these situations it is hard to debug which exactly issue happened, as
there is no stack trace in logs and no actual message from exception.
To address this issue exception is added for the logging call (rare
exception situation while stopping python worker)
### Why are the changes needed?
To effectively debug rare situations with worker termination.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
no need
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #49890 from antban/log-exception-on-worker-termination.
Lead-authored-by: dmitry.sorokin <[email protected]>
Co-authored-by: dsorokin <[email protected]>
Signed-off-by: Takuya Ueshin <[email protected]>
---
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
b/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
index 9b107cf7a3bd..84701ee593c1 100644
--- a/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
+++ b/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
@@ -274,7 +274,7 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
worker.stop()
} catch {
case e: Exception =>
- logWarning("Failed to stop worker")
+ logWarning(log"Failed to stop worker", e)
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
