This is an automated email from the ASF dual-hosted git repository.

cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 3717dc0a7477 [SPARK-54879][CORE] Add final status to Spark History 
Server for failed executions
3717dc0a7477 is described below

commit 3717dc0a747739c9959a4bdf92adc26569dd485e
Author: Rob Reeves <[email protected]>
AuthorDate: Sat Jun 27 00:32:51 2026 -0500

    [SPARK-54879][CORE] Add final status to Spark History Server for failed 
executions
    
    ### What changes were proposed in this pull request?
    This adds the final application status to SHS for failures with non-zero 
exit codes to improve debugging.
    
    #### Decisions made
    At first I wanted to always include the final exit code. When using 
spark-submit in local mode I found that throwing driver exceptions can result 
in exit code 0. This would be confusing because the status would show as a 
success. Instead, I decided to scope this to only showing the final status for 
known failures. Ideally, a future enhancement can be made to not use exit code 
0 for failed executions for all cases, but this is a riskier breaking change 
that I did not want to make here.
    
    #### Possible future extensions
    Right now I only show the exit code for explicit failures (non-zero exit 
code). There are some cases I wanted to solve, but require more foundational 
work.
    
    1. Help users understand when a driver crashed. Right now SHS will show the 
app as still running. This is another reason I didn't want to show the status 
outside of known failures. This is tricky and I wanted to infer it based on the 
time gap between the last event in the event log and current time, but that 
information is not there now (and I'm not sure on the approach).
    2. Enhance exit codes to never use 0 during a failure so it can help users 
debug more scenarios.
    3. Instead of just showing an exit code, map it to its meaning. This 
requires more work since it is resource manager specific.
    
    ### Why are the changes needed?
    If a Spark job fails due to an issue on the driver side (e.g. during the 
commit phase, custom driver code) this is not surfaced anywhere in SHS. This 
becomes even more confusing when a query/job has run successfully. SHS will 
show all jobs as completed successfully  and the only way to know the job 
failed is to look in the logs. Before this change this is what a failed driver 
side application would look like in SHS. It looks like a success.
    <img width="961" height="417" alt="before" 
src="https://github.com/user-attachments/assets/8d233666-b5cf-4268-ad2e-69184a7eb04d";
 />
    
    After the change this is what it looks like:
    <img width="1041" height="514" alt="after" 
src="https://github.com/user-attachments/assets/44b8aa13-2c85-4438-bb21-85d6b9f0c33c";
 />
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, it adds the application final status to the SHS overview when an 
execution fails and has a non-zero exit code.
    
    ### How was this patch tested?
    
    Unit tests and manually tested SHS.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code 2.0.76
    
    Closes #53657 from robreeves/shs_final_status.
    
    Authored-by: Rob Reeves <[email protected]>
    Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
---
 .../apache/spark/status/protobuf/store_types.proto |  1 +
 .../spark/deploy/history/FsHistoryProvider.scala   |  3 ++
 .../apache/spark/status/AppStatusListener.scala    |  3 +-
 .../scala/org/apache/spark/status/api/v1/api.scala |  1 +
 .../ApplicationInfoWrapperSerializer.scala         |  4 +-
 .../org/apache/spark/ui/jobs/AllJobsPage.scala     | 29 ++++++++++++
 .../application_list_json_expectation.json         | 19 ++++++++
 .../completed_app_list_json_expectation.json       | 19 ++++++++
 .../limit_app_list_json_expectation.json           | 36 ++++++++-------
 .../minDate_app_list_json_expectation.json         | 19 ++++++++
 .../minEndDate_app_list_json_expectation.json      | 19 ++++++++
 ...p_with_non_zero_exit_code_json_expectation.json | 19 ++++++++
 .../resources/spark-events/local-1430917381536     | 53 ++++++++++++++++++++++
 .../deploy/history/FsHistoryProviderSuite.scala    |  9 ++--
 .../spark/deploy/history/HistoryServerSuite.scala  |  1 +
 .../spark/status/AppStatusListenerSuite.scala      |  4 +-
 .../protobuf/KVStoreProtobufSerializerSuite.scala  |  3 +-
 dev/.rat-excludes                                  |  1 +
 project/MimaExcludes.scala                         |  3 ++
 19 files changed, 221 insertions(+), 25 deletions(-)

diff --git 
a/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto 
b/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto
index 2b6ac763cc3e..ae020e74f8c6 100644
--- a/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto
+++ b/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto
@@ -213,6 +213,7 @@ message ApplicationAttemptInfo {
   optional string spark_user = 6;
   bool completed = 7;
   optional string app_spark_version = 8;
+  optional int32 exit_code = 9;
 }
 
 message ApplicationInfo {
diff --git 
a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala 
b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
index 81830c24754e..51b0ab260dca 100644
--- 
a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
+++ 
b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
@@ -1860,6 +1860,7 @@ private[history] class AppListingListener(
     attempt.lastUpdated = new Date(reader.modificationTime)
     attempt.duration = event.time - attempt.startTime.getTime()
     attempt.completed = true
+    attempt.exitCode = event.exitCode
   }
 
   override def onEnvironmentUpdate(event: SparkListenerEnvironmentUpdate): 
Unit = {
@@ -1927,6 +1928,7 @@ private[history] class AppListingListener(
     var duration = 0L
     var sparkUser: String = null
     var completed = false
+    var exitCode = Option.empty[Int]
     var appSparkVersion = ""
 
     var adminAcls: Option[String] = None
@@ -1945,6 +1947,7 @@ private[history] class AppListingListener(
         sparkUser,
         completed,
         appSparkVersion,
+        exitCode,
         Some(logSourceName),
         Some(logSourceFullPath))
       new AttemptInfoWrapper(
diff --git 
a/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala 
b/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
index ff38d77b2b0e..7cc002b3e208 100644
--- a/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
+++ b/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
@@ -213,7 +213,8 @@ private[spark] class AppStatusListener(
       event.time - old.startTime.getTime(),
       old.sparkUser,
       true,
-      old.appSparkVersion)
+      old.appSparkVersion,
+      event.exitCode)
 
     appInfo = v1.ApplicationInfo(
       appInfo.id,
diff --git a/core/src/main/scala/org/apache/spark/status/api/v1/api.scala 
b/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
index f3fb916f0820..3e7f5e6d2ec3 100644
--- a/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
+++ b/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
@@ -54,6 +54,7 @@ case class ApplicationAttemptInfo private[spark](
     sparkUser: String,
     completed: Boolean = false,
     appSparkVersion: String,
+    exitCode: Option[Int] = None,
     logSourceName: Option[String] = None,
     logSourceFullPath: Option[String] = None) {
 
diff --git 
a/core/src/main/scala/org/apache/spark/status/protobuf/ApplicationInfoWrapperSerializer.scala
 
b/core/src/main/scala/org/apache/spark/status/protobuf/ApplicationInfoWrapperSerializer.scala
index cf3159c7b481..8526e9459472 100644
--- 
a/core/src/main/scala/org/apache/spark/status/protobuf/ApplicationInfoWrapperSerializer.scala
+++ 
b/core/src/main/scala/org/apache/spark/status/protobuf/ApplicationInfoWrapperSerializer.scala
@@ -95,6 +95,7 @@ private[protobuf] class ApplicationInfoWrapperSerializer
     info.attemptId.foreach{ id =>
       builder.setAttemptId(id)
     }
+    info.exitCode.foreach(builder.setExitCode)
     builder.build()
   }
 
@@ -110,7 +111,8 @@ private[protobuf] class ApplicationInfoWrapperSerializer
       duration = info.getDuration,
       sparkUser = getStringField(info.hasSparkUser, () => info.getSparkUser),
       completed = info.getCompleted,
-      appSparkVersion = getStringField(info.hasAppSparkVersion, () => 
info.getAppSparkVersion)
+      appSparkVersion = getStringField(info.hasAppSparkVersion, () => 
info.getAppSparkVersion),
+      exitCode = if (info.hasExitCode) Some(info.getExitCode) else None
     )
   }
 }
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala 
b/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
index 64e17a18e53b..70d0a9faf4de 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
@@ -278,6 +278,7 @@ private[ui] class AllJobsPage(parent: JobsTab, store: 
AppStatusStore) extends We
     val startDate = appInfo.attempts.head.startTime
     val startTime = startDate.getTime()
     val endTime = appInfo.attempts.head.endTime.getTime()
+    val exitCode = appInfo.attempts.head.exitCode
 
     val activeJobs = new ListBuffer[v1.JobData]()
     val completedJobs = new ListBuffer[v1.JobData]()
@@ -321,6 +322,34 @@ private[ui] class AllJobsPage(parent: JobsTab, store: 
AppStatusStore) extends We
     val summary: NodeSeq =
       <div>
         <ul class="list-unstyled">
+          <li>
+            <strong>User:</strong>
+            {parent.getSparkUser}
+          </li>
+          <li>
+            <strong>Started At:</strong>
+            {UIUtils.formatDate(startDate)}
+          </li>
+          <li>
+            <strong>Total Uptime:</strong>
+            {
+              if (endTime < 0 && parent.sc.isDefined) {
+                UIUtils.formatDuration(System.currentTimeMillis() - startTime)
+              } else if (endTime > 0) {
+                UIUtils.formatDuration(endTime - startTime)
+              }
+            }
+          </li>
+          {
+            exitCode match {
+              case Some(code) if code != 0 =>
+                <li>
+                  <strong>Exit Code:</strong>
+                  {code.toString}
+                </li>
+              case _ => <!-- -->
+            }
+          }
           <li>
             <strong>Scheduling Mode: </strong>
             {schedulingMode}
diff --git 
a/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
 
b/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
index ac6e5c63c0b6..9da4c43affe8 100644
--- 
a/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
+++ 
b/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
@@ -1,4 +1,22 @@
 [ {
+  "attempts" : [ {
+    "appSparkVersion" : "3.5.2",
+    "completed" : true,
+    "duration" : 221145,
+    "endTime" : "2026-06-23T17:34:35.148GMT",
+    "endTimeEpoch" : 1782236075148,
+    "exitCode" : 1,
+    "logSourceFullPath" : "",
+    "logSourceName" : "",
+    "lastUpdated" : "",
+    "lastUpdatedEpoch" : 0,
+    "sparkUser" : "spark",
+    "startTime" : "2026-06-23T17:30:54.003GMT",
+    "startTimeEpoch" : 1782235854003
+  } ],
+  "id" : "local-1430917381536",
+  "name" : "Spark shell"
+}, {
   "attempts" : [ {
     "appSparkVersion" : "4.2.0-SNAPSHOT",
     "logSourceFullPath" : "",
@@ -7,6 +25,7 @@
     "duration" : 2006,
     "endTime" : "2025-12-27T14:15:12.221GMT",
     "endTimeEpoch" : 1766844912221,
+    "exitCode" : 0,
     "lastUpdated" : "",
     "lastUpdatedEpoch" : 0,
     "sparkUser" : "<script>alert('XSS')</script>",
diff --git 
a/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
 
b/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
index ac6e5c63c0b6..9da4c43affe8 100644
--- 
a/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
+++ 
b/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
@@ -1,4 +1,22 @@
 [ {
+  "attempts" : [ {
+    "appSparkVersion" : "3.5.2",
+    "completed" : true,
+    "duration" : 221145,
+    "endTime" : "2026-06-23T17:34:35.148GMT",
+    "endTimeEpoch" : 1782236075148,
+    "exitCode" : 1,
+    "logSourceFullPath" : "",
+    "logSourceName" : "",
+    "lastUpdated" : "",
+    "lastUpdatedEpoch" : 0,
+    "sparkUser" : "spark",
+    "startTime" : "2026-06-23T17:30:54.003GMT",
+    "startTimeEpoch" : 1782235854003
+  } ],
+  "id" : "local-1430917381536",
+  "name" : "Spark shell"
+}, {
   "attempts" : [ {
     "appSparkVersion" : "4.2.0-SNAPSHOT",
     "logSourceFullPath" : "",
@@ -7,6 +25,7 @@
     "duration" : 2006,
     "endTime" : "2025-12-27T14:15:12.221GMT",
     "endTimeEpoch" : 1766844912221,
+    "exitCode" : 0,
     "lastUpdated" : "",
     "lastUpdatedEpoch" : 0,
     "sparkUser" : "<script>alert('XSS')</script>",
diff --git 
a/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
 
b/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
index 06ef3a5ecc9f..fef54a1faa3f 100644
--- 
a/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
+++ 
b/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
@@ -1,4 +1,22 @@
 [ {
+  "attempts" : [ {
+    "appSparkVersion" : "3.5.2",
+    "completed" : true,
+    "duration" : 221145,
+    "endTime" : "2026-06-23T17:34:35.148GMT",
+    "endTimeEpoch" : 1782236075148,
+    "exitCode" : 1,
+    "logSourceFullPath" : "",
+    "logSourceName" : "",
+    "lastUpdated" : "",
+    "lastUpdatedEpoch" : 0,
+    "sparkUser" : "spark",
+    "startTime" : "2026-06-23T17:30:54.003GMT",
+    "startTimeEpoch" : 1782235854003
+  } ],
+  "id" : "local-1430917381536",
+  "name" : "Spark shell"
+}, {
   "attempts" : [ {
     "appSparkVersion" : "4.2.0-SNAPSHOT",
     "logSourceFullPath" : "",
@@ -7,6 +25,7 @@
     "duration" : 2006,
     "endTime" : "2025-12-27T14:15:12.221GMT",
     "endTimeEpoch" : 1766844912221,
+    "exitCode" : 0,
     "lastUpdated" : "",
     "lastUpdatedEpoch" : 0,
     "sparkUser" : "<script>alert('XSS')</script>",
@@ -32,21 +51,4 @@
   } ],
   "id" : "local-1642039451826",
   "name" : "Spark shell"
-}, {
-  "attempts" : [ {
-    "appSparkVersion" : "3.1.1.119",
-    "logSourceFullPath" : "",
-    "logSourceName" : "",
-    "completed" : true,
-    "duration" : 274875,
-    "endTime" : "2021-08-10T23:29:30.208GMT",
-    "endTimeEpoch" : 1628638170208,
-    "lastUpdated" : "",
-    "lastUpdatedEpoch" : 0,
-    "sparkUser" : "john",
-    "startTime" : "2021-08-10T23:24:55.333GMT",
-    "startTimeEpoch" : 1628637895333
-  } ],
-  "id" : "application_1628109047826_1317105",
-  "name" : "Spark shell"
 } ]
diff --git 
a/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
 
b/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
index a1d2f9e9ef4d..885045f2faf5 100644
--- 
a/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
+++ 
b/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
@@ -1,4 +1,22 @@
 [ {
+  "attempts" : [ {
+    "appSparkVersion" : "3.5.2",
+    "completed" : true,
+    "duration" : 221145,
+    "endTime" : "2026-06-23T17:34:35.148GMT",
+    "endTimeEpoch" : 1782236075148,
+    "exitCode" : 1,
+    "logSourceFullPath" : "",
+    "logSourceName" : "",
+    "lastUpdated" : "",
+    "lastUpdatedEpoch" : 0,
+    "sparkUser" : "spark",
+    "startTime" : "2026-06-23T17:30:54.003GMT",
+    "startTimeEpoch" : 1782235854003
+  } ],
+  "id" : "local-1430917381536",
+  "name" : "Spark shell"
+}, {
   "attempts" : [ {
     "appSparkVersion" : "4.2.0-SNAPSHOT",
     "logSourceFullPath" : "",
@@ -7,6 +25,7 @@
     "duration" : 2006,
     "endTime" : "2025-12-27T14:15:12.221GMT",
     "endTimeEpoch" : 1766844912221,
+    "exitCode" : 0,
     "lastUpdated" : "",
     "lastUpdatedEpoch" : 0,
     "sparkUser" : "<script>alert('XSS')</script>",
diff --git 
a/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
 
b/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
index 413660c4b689..ead957a5fbbb 100644
--- 
a/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
+++ 
b/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
@@ -1,4 +1,22 @@
 [ {
+  "attempts" : [ {
+    "appSparkVersion" : "3.5.2",
+    "completed" : true,
+    "duration" : 221145,
+    "endTime" : "2026-06-23T17:34:35.148GMT",
+    "endTimeEpoch" : 1782236075148,
+    "exitCode" : 1,
+    "logSourceFullPath" : "",
+    "logSourceName" : "",
+    "lastUpdated" : "",
+    "lastUpdatedEpoch" : 0,
+    "sparkUser" : "spark",
+    "startTime" : "2026-06-23T17:30:54.003GMT",
+    "startTimeEpoch" : 1782235854003
+  } ],
+  "id" : "local-1430917381536",
+  "name" : "Spark shell"
+}, {
   "attempts" : [ {
     "appSparkVersion" : "4.2.0-SNAPSHOT",
     "logSourceFullPath" : "",
@@ -7,6 +25,7 @@
     "duration" : 2006,
     "endTime" : "2025-12-27T14:15:12.221GMT",
     "endTimeEpoch" : 1766844912221,
+    "exitCode" : 0,
     "lastUpdated" : "",
     "lastUpdatedEpoch" : 0,
     "sparkUser" : "<script>alert('XSS')</script>",
diff --git 
a/core/src/test/resources/HistoryServerExpectations/one_app_with_non_zero_exit_code_json_expectation.json
 
b/core/src/test/resources/HistoryServerExpectations/one_app_with_non_zero_exit_code_json_expectation.json
new file mode 100644
index 000000000000..9b9ee4c7ed9d
--- /dev/null
+++ 
b/core/src/test/resources/HistoryServerExpectations/one_app_with_non_zero_exit_code_json_expectation.json
@@ -0,0 +1,19 @@
+{
+  "attempts" : [ {
+    "appSparkVersion" : "3.5.2",
+    "completed" : true,
+    "duration" : 221145,
+    "endTime" : "2026-06-23T17:34:35.148GMT",
+    "endTimeEpoch" : 1782236075148,
+    "exitCode" : 1,
+    "logSourceFullPath" : "",
+    "logSourceName" : "",
+    "lastUpdated" : "",
+    "lastUpdatedEpoch" : 0,
+    "sparkUser" : "spark",
+    "startTime" : "2026-06-23T17:30:54.003GMT",
+    "startTimeEpoch" : 1782235854003
+  } ],
+  "id" : "local-1430917381536",
+  "name" : "Spark shell"
+}
diff --git a/core/src/test/resources/spark-events/local-1430917381536 
b/core/src/test/resources/spark-events/local-1430917381536
new file mode 100644
index 000000000000..c356d307895e
--- /dev/null
+++ b/core/src/test/resources/spark-events/local-1430917381536
@@ -0,0 +1,53 @@
+{"Event":"SparkListenerLogStart","Spark Version":"3.5.2"}
+{"Event":"SparkListenerResourceProfileAdded","Resource Profile Id":0,"Executor 
Resource Requests":{"cores":{"Resource Name":"cores","Amount":1,"Discovery 
Script":"","Vendor":""},"memory":{"Resource 
Name":"memory","Amount":2048,"Discovery 
Script":"","Vendor":""},"offHeap":{"Resource 
Name":"offHeap","Amount":0,"Discovery Script":"","Vendor":""}},"Task Resource 
Requests":{"cpus":{"Resource Name":"cpus","Amount":1.0}}}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":587552240,"JVMOffHeapMemory":169149216,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":41075,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9117978624,"ProcessTreeJVMRSSMemory":1213
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":587552240,"JVMOffHeapMemory":169436864,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":41075,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9117978624,"ProcessTreeJVMRSSMemory":1213
 [...]
+{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor 
ID":"driver","Host":"localhost","Port":24790},"Maximum 
Memory":3676726886,"Timestamp":1782235880552,"Maximum Onheap 
Memory":3676726886,"Maximum Offheap Memory":0}
+{"Event":"SparkListenerEnvironmentUpdate","JVM Information":{"Java 
Home":"/usr/lib/jvm/java","Java Version":"17.0.5 (Microsoft)","Scala 
Version":"version 2.12.18"},"Spark Properties":{},"Hadoop 
Properties":{},"System Properties":{},"Metrics Properties":{},"Classpath 
Entries":{}}
+{"Event":"SparkListenerApplicationStart","App Name":"Spark shell","App 
ID":"local-1430917381536","Timestamp":1782235854003,"User":"spark"}
+{"Event":"org.apache.spark.scheduler.SparkListenerMiscellaneousProcessAdded","time":1782235881468,"processId":"yarn-am","info":{"hostPort":"localhost:8042","cores":1,"logUrlInfo":{"stdout":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000001/spark/stdout?start=-4096","stderr":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000001/spark/stderr?start=-4096"}}}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":663049712,"JVMOffHeapMemory":181725520,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":4311030,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9135951872,"ProcessTreeJVMRSSMemory":12
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":663049712,"JVMOffHeapMemory":182043856,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":4311030,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9135951872,"ProcessTreeJVMRSSMemory":12
 [...]
+{"Event":"SparkListenerExecutorAdded","Timestamp":1782235902754,"Executor 
ID":"1","Executor Info":{"Host":"localhost","Total Cores":1,"Log 
Urls":{"stdout":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000002/spark/stdout?start=-4096","stderr":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000002/spark/stderr?start=-4096"},"Attributes":{"NM_HTTP_ADDRESS":"localhost:8042","USER":"spark","LOG_FILES":"stderr,stdout","NM_HTTP_PORT":"8042";
 [...]
+{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor 
ID":"1","Host":"localhost","Port":11366},"Maximum 
Memory":1099746508,"Timestamp":1782235902912,"Maximum Onheap 
Memory":1099746508,"Maximum Offheap Memory":0}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":684021232,"JVMOffHeapMemory":183676472,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":12723107,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9146445824,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerExecutorAdded","Timestamp":1782235908831,"Executor 
ID":"3","Executor Info":{"Host":"localhost","Total Cores":1,"Log 
Urls":{"stdout":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000003/spark/stdout?start=-4096","stderr":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000003/spark/stderr?start=-4096"},"Attributes":{"NM_HTTP_ADDRESS":"localhost:8042","USER":"spark","LOG_FILES":"stderr,stdout","NM_HTTP_PORT":"8042";
 [...]
+{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor 
ID":"3","Host":"localhost","Port":11298},"Maximum 
Memory":1099746508,"Timestamp":1782235908994,"Maximum Onheap 
Memory":1099746508,"Maximum Offheap Memory":0}
+{"Event":"SparkListenerExecutorAdded","Timestamp":1782235910397,"Executor 
ID":"2","Executor Info":{"Host":"localhost","Total Cores":1,"Log 
Urls":{"stdout":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000004/spark/stdout?start=-4096","stderr":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000004/spark/stderr?start=-4096"},"Attributes":{"NM_HTTP_ADDRESS":"localhost:8042","USER":"spark","LOG_FILES":"stderr,stdout","NM_HTTP_PORT":"8042";
 [...]
+{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor 
ID":"2","Host":"localhost","Port":16966},"Maximum 
Memory":1099746508,"Timestamp":1782235910574,"Maximum Onheap 
Memory":1099746508,"Maximum Offheap Memory":0}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":500075000,"JVMOffHeapMemory":200249712,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29547261,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9211654144,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":562989560,"JVMOffHeapMemory":207438768,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29659885,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9231601664,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":528762488,"JVMOffHeapMemory":230684216,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29659885,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9231601664,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":532956792,"JVMOffHeapMemory":230282752,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29668077,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9231601664,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":537151096,"JVMOffHeapMemory":232446888,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29668077,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9239990272,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":709117560,"JVMOffHeapMemory":244835032,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29816850,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9239990272,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":713311864,"JVMOffHeapMemory":244563024,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29816850,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9240129536,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerExecutorRemoved","Timestamp":1782235983810,"Executor 
ID":"1","Removed Reason":"Executor decommission: spark scale down"}
+{"Event":"SparkListenerBlockManagerRemoved","Block Manager ID":{"Executor 
ID":"1","Host":"localhost","Port":11366},"Timestamp":1782235983814}
+{"Event":"SparkListenerExecutorRemoved","Timestamp":1782235985799,"Executor 
ID":"1","Removed Reason":"Executor for container 
container_e121_1430917381536_01_000002 exited because of a YARN event (e.g., 
preemption) and not because of an error in the running job."}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":793003640,"JVMOffHeapMemory":248284608,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29808921,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerExecutorRemoved","Timestamp":1782235989914,"Executor 
ID":"3","Removed Reason":"Executor decommission: spark scale down"}
+{"Event":"SparkListenerBlockManagerRemoved","Block Manager ID":{"Executor 
ID":"3","Host":"localhost","Port":11298},"Timestamp":1782235989915}
+{"Event":"SparkListenerExecutorRemoved","Timestamp":1782235991801,"Executor 
ID":"3","Removed Reason":"Executor for container 
container_e121_1430917381536_01_000003 exited because of a YARN event (e.g., 
preemption) and not because of an error in the running job."}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":801392248,"JVMOffHeapMemory":249805880,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29808922,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":805586552,"JVMOffHeapMemory":249421632,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29800730,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":805586552,"JVMOffHeapMemory":249945392,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29800730,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":809780856,"JVMOffHeapMemory":250125488,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29780056,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerJobStart","Job ID":0,"Submission 
Time":1782236034538,"Stage Infos":[{"Stage ID":0,"Stage Attempt ID":0,"Stage 
Name":"writeDataFrame at <console>:84","Number of Tasks":3,"RDD Info":[{"RDD 
ID":3,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"0\",\"name\":\"WriteFiles\"}","Callsite":"writeDataFrame
 at <console>:84","Parent IDs":[2],"Storage Level":{"Use Disk":false,"Use 
Memory":false,"Use Off 
Heap":false,"Deserialized":false,"Replication":1},"Barrier":false,"Determinist 
[...]
+{"Event":"SparkListenerStageSubmitted","Stage Info":{"Stage ID":0,"Stage 
Attempt ID":0,"Stage Name":"writeDataFrame at <console>:84","Number of 
Tasks":3,"RDD Info":[{"RDD 
ID":3,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"0\",\"name\":\"WriteFiles\"}","Callsite":"writeDataFrame
 at <console>:84","Parent IDs":[2],"Storage Level":{"Use Disk":false,"Use 
Memory":false,"Use Off 
Heap":false,"Deserialized":false,"Replication":1},"Barrier":false,"DeterministicLevel":"DETERMINATE","Number
 of Parti [...]
+{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task 
Info":{"Task ID":0,"Index":0,"Attempt":0,"Partition ID":0,"Launch 
Time":1782236034680,"Executor 
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Resource Profile 
ID":0,"Accumulables":[]}}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":673064904,"JVMOffHeapMemory":281199496,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33969867,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9349148672,"ProcessTreeJVMRS
 [...]
+{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task 
Info":{"Task ID":1,"Index":1,"Attempt":0,"Partition ID":1,"Launch 
Time":1782236037542,"Executor 
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Resource Profile 
ID":0,"Accumulables":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":0,"Index":0,"Attempt":0,"Partition ID":0,"Launch 
Time":1782236034680,"Executor 
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1782236037555,"Failed":false,"Killed":false,"Resource Profile 
ID":0,"Accumulables":[{"ID":4,"Name":"task commit 
time","Update":"50","Value":"50","In [...]
+{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task 
Info":{"Task ID":2,"Index":2,"Attempt":0,"Partition ID":2,"Launch 
Time":1782236037693,"Executor 
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Resource Profile 
ID":0,"Accumulables":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":1,"Index":1,"Attempt":0,"Partition ID":1,"Launch 
Time":1782236037542,"Executor 
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1782236037694,"Failed":false,"Killed":false,"Resource Profile 
ID":0,"Accumulables":[{"ID":4,"Name":"task commit 
time","Update":"37","Value":"87","In [...]
+{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task 
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task 
ID":2,"Index":2,"Attempt":0,"Partition ID":2,"Launch 
Time":1782236037693,"Executor 
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
 Result Time":0,"Finish 
Time":1782236037801,"Failed":false,"Killed":false,"Resource Profile 
ID":0,"Accumulables":[{"ID":4,"Name":"task commit 
time","Update":"40","Value":"127","I [...]
+{"Event":"SparkListenerStageExecutorMetrics","Executor ID":"2","Stage 
ID":0,"Stage Attempt ID":0,"Executor 
Metrics":{"JVMHeapMemory":186850048,"JVMOffHeapMemory":116149944,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":16807195,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":4315377664,"ProcessTreeJVMRSSMemory":628396032,"ProcessTreePythonVMemory":0,"ProcessTr
 [...]
+{"Event":"SparkListenerStageExecutorMetrics","Executor ID":"driver","Stage 
ID":0,"Stage Attempt ID":0,"Executor 
Metrics":{"JVMHeapMemory":673064904,"JVMOffHeapMemory":281199496,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33969867,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9349148672,"ProcessTreeJVMRSSMemory":1489293312,"ProcessTreePythonVMemory":0,"Pro
 [...]
+{"Event":"SparkListenerStageCompleted","Stage Info":{"Stage ID":0,"Stage 
Attempt ID":0,"Stage Name":"writeDataFrame at <console>:84","Number of 
Tasks":3,"RDD Info":[{"RDD 
ID":3,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"0\",\"name\":\"WriteFiles\"}","Callsite":"writeDataFrame
 at <console>:84","Parent IDs":[2],"Storage Level":{"Use Disk":false,"Use 
Memory":false,"Use Off 
Heap":false,"Deserialized":false,"Replication":1},"Barrier":false,"DeterministicLevel":"DETERMINATE","Number
 of Parti [...]
+{"Event":"SparkListenerJobEnd","Job ID":0,"Completion Time":1782236037805,"Job 
Result":{"Result":"JobSucceeded"}}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":798894024,"JVMOffHeapMemory":302532712,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33973602,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9373675520,"ProcessTreeJVMRS
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":807282632,"JVMOffHeapMemory":304663000,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33991159,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9373675520,"ProcessTreeJVMRS
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":811476936,"JVMOffHeapMemory":306466816,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33999351,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9373675520,"ProcessTreeJVMRS
 [...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics 
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt 
ID":-1,"Executor 
Metrics":{"JVMHeapMemory":542426624,"JVMOffHeapMemory":305486288,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33999351,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9391607808,"ProcessTreeJVMRSSMemory":1
 [...]
+{"Event":"SparkListenerApplicationEnd","Timestamp":1782236075148,"ExitCode":1}
diff --git 
a/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
 
b/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
index 8a4fd0ded959..01a77b880ebc 100644
--- 
a/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
+++ 
b/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
@@ -122,7 +122,7 @@ abstract class FsHistoryProviderSuite extends SparkFunSuite 
with Matchers with P
     writeFile(newAppComplete, None,
       SparkListenerApplicationStart(newAppComplete.getName(), 
Some("new-app-complete"), 1L, "test",
         None),
-      SparkListenerApplicationEnd(5L)
+      SparkListenerApplicationEnd(5L, Some(1))
       )
 
     // Write a new-style application log.
@@ -154,7 +154,8 @@ abstract class FsHistoryProviderSuite extends SparkFunSuite 
with Matchers with P
           end: Long,
           lastMod: Long,
           user: String,
-          completed: Boolean): ApplicationInfo = {
+          completed: Boolean,
+          exitCode: Option[Int] = None): ApplicationInfo = {
 
         val duration = if (end > 0) end - start else 0
         // Get log dir info from the provider
@@ -162,13 +163,13 @@ abstract class FsHistoryProviderSuite extends 
SparkFunSuite with Matchers with P
         val (logSourceName, logSourceFullPath) = 
provider.getLogDirInfo(logPath)
         new ApplicationInfo(id, name, None, None, None, None,
           List(ApplicationAttemptInfo(None, new Date(start),
-            new Date(end), new Date(lastMod), duration, user, completed, 
SPARK_VERSION,
+            new Date(end), new Date(lastMod), duration, user, completed, 
SPARK_VERSION, exitCode,
             Some(logSourceName), Some(logSourceFullPath))))
       }
 
       // For completed files, lastUpdated would be lastModified time.
       list(0) should be (makeAppInfo("new-app-complete", 
newAppComplete.getName(), 1L, 5L,
-        newAppComplete.lastModified(), "test", true))
+        newAppComplete.lastModified(), "test", true, Some(1)))
       list(1) should be (makeAppInfo("new-complete-lzf", 
newAppCompressedComplete.getName(),
         1L, 4L, newAppCompressedComplete.lastModified(), "test", true))
 
diff --git 
a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala 
b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
index 13432b6ed9fc..79e0001fa22f 100644
--- 
a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
+++ 
b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
@@ -140,6 +140,7 @@ abstract class HistoryServerSuite extends SparkFunSuite 
with BeforeAndAfter with
     "limit app list json" -> "applications?limit=3",
     "one app json" -> "applications/local-1422981780767",
     "one app multi-attempt json" -> "applications/local-1426533911241",
+    "one app with non-zero exit code json" -> 
"applications/local-1430917381536",
     "job list json" -> "applications/local-1422981780767/jobs",
     "job list from multi-attempt app json(1)" -> 
"applications/local-1426533911241/1/jobs",
     "job list from multi-attempt app json(2)" -> 
"applications/local-1426533911241/2/jobs",
diff --git 
a/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala 
b/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
index 193c2a9f3349..e0a6120b6a59 100644
--- a/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
@@ -200,6 +200,7 @@ abstract class AppStatusListenerSuite extends SparkFunSuite 
with BeforeAndAfter
       assert(attempt.endTime.getTime() === -1L)
       assert(attempt.sparkUser === "user")
       assert(!attempt.completed)
+      assert(attempt.exitCode === None)
       assert(attempt.appSparkVersion === "TestSparkVersion")
     }
 
@@ -757,7 +758,7 @@ abstract class AppStatusListenerSuite extends SparkFunSuite 
with BeforeAndAfter
     }
 
     // End the application.
-    listener.onApplicationEnd(SparkListenerApplicationEnd(42L))
+    listener.onApplicationEnd(SparkListenerApplicationEnd(42L, Some(1)))
 
     check[ApplicationInfoWrapper]("id") { app =>
       assert(app.info.name === "name")
@@ -772,6 +773,7 @@ abstract class AppStatusListenerSuite extends SparkFunSuite 
with BeforeAndAfter
       assert(attempt.duration === 41L)
       assert(attempt.sparkUser === "user")
       assert(attempt.completed)
+      assert(attempt.exitCode === Some(1))
     }
   }
 
diff --git 
a/core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala
 
b/core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala
index 7bb36a97163b..23cb99bce2ff 100644
--- 
a/core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala
+++ 
b/core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala
@@ -399,7 +399,8 @@ class KVStoreProtobufSerializerSuite extends SparkFunSuite {
         duration = 100,
         sparkUser = null,
         completed = true,
-        appSparkVersion = null
+        appSparkVersion = null,
+        exitCode = Some(1)
       ))
     val input = new ApplicationInfoWrapper(
       ApplicationInfo(
diff --git a/dev/.rat-excludes b/dev/.rat-excludes
index 2eca24afcec5..4aee210e85b9 100644
--- a/dev/.rat-excludes
+++ b/dev/.rat-excludes
@@ -84,6 +84,7 @@ local-1426633911242
 local-1430917381534
 local-1430917381535_1
 local-1430917381535_2
+local-1430917381536
 local-1642039451826
 eventlog_v2_local-1766844910796
 DESCRIPTION
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index 984ce2f37ce5..cfd64334ae91 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -59,6 +59,9 @@ object MimaExcludes {
 
   // Exclude rules for 4.2.x from 4.1.0
   lazy val v42excludes = v41excludes ++ Seq(
+    // [SPARK-54879] Add exitCode field to ApplicationAttemptInfo
+    
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.ApplicationAttemptInfo.tupled"),
+    
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.ApplicationAttemptInfo.curried"),
     // [SQL] SafeJsonSerializer.safeMapToJValue: second parameter widened from 
Function1 to
     // Function2 so the key is passed to the value serializer 
(progress.scala). Binary-incompatible
     // vs spark-sql-api 4.0.0; not part of the public supported API 
(private[streaming] package).


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to