This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e7709f0f60e5 [SPARK-51155][CORE] Make `SparkContext` show total
runtime after stopping
e7709f0f60e5 is described below
commit e7709f0f60e5fa13cd1d494d6cc6f6e8fc7988ff
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Feb 10 23:03:19 2025 -0800
[SPARK-51155][CORE] Make `SparkContext` show total runtime after stopping
### What changes were proposed in this pull request?
This PR aims to make `SparkContext` show total runtime after stopping.
### Why are the changes needed?
In Spark UI, we can check `Total Uptime`.
<img width="275" alt="Screenshot 2025-02-10 at 21 41 48"
src="https://github.com/user-attachments/assets/d31ede44-ac07-428e-b26d-4c90cfda17d2";
/>
It would be great if the log shows the total uptime of `SparkContext`.
**BEFORE**
```
$ bin/run-example SparkPi 2>&1 | grep SparkContext | tail -n1
25/02/10 21:41:13 INFO SparkContext: Successfully stopped SparkContext
```
**AFTER**
```
$ bin/run-example SparkPi 2>&1 | grep SparkContext | tail -n1
25/02/10 21:39:34 INFO SparkContext: Successfully stopped SparkContext
(Uptime: 3453 ms)
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually run the job and check the log because this is a log-related PR.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #49878 from dongjoon-hyun/SPARK-51155.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/SparkContext.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 1b27a0fbc711..143a0d5e633c 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -2405,7 +2405,8 @@ class SparkContext(config: SparkConf) extends Logging {
ResourceProfile.clearDefaultProfile()
// Unset YARN mode system env variable, to allow switching between cluster
types.
SparkContext.clearActiveContext()
- logInfo("Successfully stopped SparkContext")
+ logInfo(log"Successfully stopped SparkContext (Uptime: " +
+ log"${MDC(LogKeys.TOTAL_TIME, System.currentTimeMillis() - startTime)}
ms)")
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
