This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 3717dc0a7477 [SPARK-54879][CORE] Add final status to Spark History
Server for failed executions
3717dc0a7477 is described below
commit 3717dc0a747739c9959a4bdf92adc26569dd485e
Author: Rob Reeves <[email protected]>
AuthorDate: Sat Jun 27 00:32:51 2026 -0500
[SPARK-54879][CORE] Add final status to Spark History Server for failed
executions
### What changes were proposed in this pull request?
This adds the final application status to SHS for failures with non-zero
exit codes to improve debugging.
#### Decisions made
At first I wanted to always include the final exit code. When using
spark-submit in local mode I found that throwing driver exceptions can result
in exit code 0. This would be confusing because the status would show as a
success. Instead, I decided to scope this to only showing the final status for
known failures. Ideally, a future enhancement can be made to not use exit code
0 for failed executions for all cases, but this is a riskier breaking change
that I did not want to make here.
#### Possible future extensions
Right now I only show the exit code for explicit failures (non-zero exit
code). There are some cases I wanted to solve, but require more foundational
work.
1. Help users understand when a driver crashed. Right now SHS will show the
app as still running. This is another reason I didn't want to show the status
outside of known failures. This is tricky and I wanted to infer it based on the
time gap between the last event in the event log and current time, but that
information is not there now (and I'm not sure on the approach).
2. Enhance exit codes to never use 0 during a failure so it can help users
debug more scenarios.
3. Instead of just showing an exit code, map it to its meaning. This
requires more work since it is resource manager specific.
### Why are the changes needed?
If a Spark job fails due to an issue on the driver side (e.g. during the
commit phase, custom driver code) this is not surfaced anywhere in SHS. This
becomes even more confusing when a query/job has run successfully. SHS will
show all jobs as completed successfully and the only way to know the job
failed is to look in the logs. Before this change this is what a failed driver
side application would look like in SHS. It looks like a success.
<img width="961" height="417" alt="before"
src="https://github.com/user-attachments/assets/8d233666-b5cf-4268-ad2e-69184a7eb04d"
/>
After the change this is what it looks like:
<img width="1041" height="514" alt="after"
src="https://github.com/user-attachments/assets/44b8aa13-2c85-4438-bb21-85d6b9f0c33c"
/>
### Does this PR introduce _any_ user-facing change?
Yes, it adds the application final status to the SHS overview when an
execution fails and has a non-zero exit code.
### How was this patch tested?
Unit tests and manually tested SHS.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code 2.0.76
Closes #53657 from robreeves/shs_final_status.
Authored-by: Rob Reeves <[email protected]>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
---
.../apache/spark/status/protobuf/store_types.proto | 1 +
.../spark/deploy/history/FsHistoryProvider.scala | 3 ++
.../apache/spark/status/AppStatusListener.scala | 3 +-
.../scala/org/apache/spark/status/api/v1/api.scala | 1 +
.../ApplicationInfoWrapperSerializer.scala | 4 +-
.../org/apache/spark/ui/jobs/AllJobsPage.scala | 29 ++++++++++++
.../application_list_json_expectation.json | 19 ++++++++
.../completed_app_list_json_expectation.json | 19 ++++++++
.../limit_app_list_json_expectation.json | 36 ++++++++-------
.../minDate_app_list_json_expectation.json | 19 ++++++++
.../minEndDate_app_list_json_expectation.json | 19 ++++++++
...p_with_non_zero_exit_code_json_expectation.json | 19 ++++++++
.../resources/spark-events/local-1430917381536 | 53 ++++++++++++++++++++++
.../deploy/history/FsHistoryProviderSuite.scala | 9 ++--
.../spark/deploy/history/HistoryServerSuite.scala | 1 +
.../spark/status/AppStatusListenerSuite.scala | 4 +-
.../protobuf/KVStoreProtobufSerializerSuite.scala | 3 +-
dev/.rat-excludes | 1 +
project/MimaExcludes.scala | 3 ++
19 files changed, 221 insertions(+), 25 deletions(-)
diff --git
a/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto
b/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto
index 2b6ac763cc3e..ae020e74f8c6 100644
--- a/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto
+++ b/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto
@@ -213,6 +213,7 @@ message ApplicationAttemptInfo {
optional string spark_user = 6;
bool completed = 7;
optional string app_spark_version = 8;
+ optional int32 exit_code = 9;
}
message ApplicationInfo {
diff --git
a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
index 81830c24754e..51b0ab260dca 100644
---
a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
@@ -1860,6 +1860,7 @@ private[history] class AppListingListener(
attempt.lastUpdated = new Date(reader.modificationTime)
attempt.duration = event.time - attempt.startTime.getTime()
attempt.completed = true
+ attempt.exitCode = event.exitCode
}
override def onEnvironmentUpdate(event: SparkListenerEnvironmentUpdate):
Unit = {
@@ -1927,6 +1928,7 @@ private[history] class AppListingListener(
var duration = 0L
var sparkUser: String = null
var completed = false
+ var exitCode = Option.empty[Int]
var appSparkVersion = ""
var adminAcls: Option[String] = None
@@ -1945,6 +1947,7 @@ private[history] class AppListingListener(
sparkUser,
completed,
appSparkVersion,
+ exitCode,
Some(logSourceName),
Some(logSourceFullPath))
new AttemptInfoWrapper(
diff --git
a/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
b/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
index ff38d77b2b0e..7cc002b3e208 100644
--- a/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
+++ b/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
@@ -213,7 +213,8 @@ private[spark] class AppStatusListener(
event.time - old.startTime.getTime(),
old.sparkUser,
true,
- old.appSparkVersion)
+ old.appSparkVersion,
+ event.exitCode)
appInfo = v1.ApplicationInfo(
appInfo.id,
diff --git a/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
b/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
index f3fb916f0820..3e7f5e6d2ec3 100644
--- a/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
+++ b/core/src/main/scala/org/apache/spark/status/api/v1/api.scala
@@ -54,6 +54,7 @@ case class ApplicationAttemptInfo private[spark](
sparkUser: String,
completed: Boolean = false,
appSparkVersion: String,
+ exitCode: Option[Int] = None,
logSourceName: Option[String] = None,
logSourceFullPath: Option[String] = None) {
diff --git
a/core/src/main/scala/org/apache/spark/status/protobuf/ApplicationInfoWrapperSerializer.scala
b/core/src/main/scala/org/apache/spark/status/protobuf/ApplicationInfoWrapperSerializer.scala
index cf3159c7b481..8526e9459472 100644
---
a/core/src/main/scala/org/apache/spark/status/protobuf/ApplicationInfoWrapperSerializer.scala
+++
b/core/src/main/scala/org/apache/spark/status/protobuf/ApplicationInfoWrapperSerializer.scala
@@ -95,6 +95,7 @@ private[protobuf] class ApplicationInfoWrapperSerializer
info.attemptId.foreach{ id =>
builder.setAttemptId(id)
}
+ info.exitCode.foreach(builder.setExitCode)
builder.build()
}
@@ -110,7 +111,8 @@ private[protobuf] class ApplicationInfoWrapperSerializer
duration = info.getDuration,
sparkUser = getStringField(info.hasSparkUser, () => info.getSparkUser),
completed = info.getCompleted,
- appSparkVersion = getStringField(info.hasAppSparkVersion, () =>
info.getAppSparkVersion)
+ appSparkVersion = getStringField(info.hasAppSparkVersion, () =>
info.getAppSparkVersion),
+ exitCode = if (info.hasExitCode) Some(info.getExitCode) else None
)
}
}
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
b/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
index 64e17a18e53b..70d0a9faf4de 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
@@ -278,6 +278,7 @@ private[ui] class AllJobsPage(parent: JobsTab, store:
AppStatusStore) extends We
val startDate = appInfo.attempts.head.startTime
val startTime = startDate.getTime()
val endTime = appInfo.attempts.head.endTime.getTime()
+ val exitCode = appInfo.attempts.head.exitCode
val activeJobs = new ListBuffer[v1.JobData]()
val completedJobs = new ListBuffer[v1.JobData]()
@@ -321,6 +322,34 @@ private[ui] class AllJobsPage(parent: JobsTab, store:
AppStatusStore) extends We
val summary: NodeSeq =
<div>
<ul class="list-unstyled">
+ <li>
+ <strong>User:</strong>
+ {parent.getSparkUser}
+ </li>
+ <li>
+ <strong>Started At:</strong>
+ {UIUtils.formatDate(startDate)}
+ </li>
+ <li>
+ <strong>Total Uptime:</strong>
+ {
+ if (endTime < 0 && parent.sc.isDefined) {
+ UIUtils.formatDuration(System.currentTimeMillis() - startTime)
+ } else if (endTime > 0) {
+ UIUtils.formatDuration(endTime - startTime)
+ }
+ }
+ </li>
+ {
+ exitCode match {
+ case Some(code) if code != 0 =>
+ <li>
+ <strong>Exit Code:</strong>
+ {code.toString}
+ </li>
+ case _ => <!-- -->
+ }
+ }
<li>
<strong>Scheduling Mode: </strong>
{schedulingMode}
diff --git
a/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
index ac6e5c63c0b6..9da4c43affe8 100644
---
a/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
+++
b/core/src/test/resources/HistoryServerExpectations/application_list_json_expectation.json
@@ -1,4 +1,22 @@
[ {
+ "attempts" : [ {
+ "appSparkVersion" : "3.5.2",
+ "completed" : true,
+ "duration" : 221145,
+ "endTime" : "2026-06-23T17:34:35.148GMT",
+ "endTimeEpoch" : 1782236075148,
+ "exitCode" : 1,
+ "logSourceFullPath" : "",
+ "logSourceName" : "",
+ "lastUpdated" : "",
+ "lastUpdatedEpoch" : 0,
+ "sparkUser" : "spark",
+ "startTime" : "2026-06-23T17:30:54.003GMT",
+ "startTimeEpoch" : 1782235854003
+ } ],
+ "id" : "local-1430917381536",
+ "name" : "Spark shell"
+}, {
"attempts" : [ {
"appSparkVersion" : "4.2.0-SNAPSHOT",
"logSourceFullPath" : "",
@@ -7,6 +25,7 @@
"duration" : 2006,
"endTime" : "2025-12-27T14:15:12.221GMT",
"endTimeEpoch" : 1766844912221,
+ "exitCode" : 0,
"lastUpdated" : "",
"lastUpdatedEpoch" : 0,
"sparkUser" : "<script>alert('XSS')</script>",
diff --git
a/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
index ac6e5c63c0b6..9da4c43affe8 100644
---
a/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
+++
b/core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
@@ -1,4 +1,22 @@
[ {
+ "attempts" : [ {
+ "appSparkVersion" : "3.5.2",
+ "completed" : true,
+ "duration" : 221145,
+ "endTime" : "2026-06-23T17:34:35.148GMT",
+ "endTimeEpoch" : 1782236075148,
+ "exitCode" : 1,
+ "logSourceFullPath" : "",
+ "logSourceName" : "",
+ "lastUpdated" : "",
+ "lastUpdatedEpoch" : 0,
+ "sparkUser" : "spark",
+ "startTime" : "2026-06-23T17:30:54.003GMT",
+ "startTimeEpoch" : 1782235854003
+ } ],
+ "id" : "local-1430917381536",
+ "name" : "Spark shell"
+}, {
"attempts" : [ {
"appSparkVersion" : "4.2.0-SNAPSHOT",
"logSourceFullPath" : "",
@@ -7,6 +25,7 @@
"duration" : 2006,
"endTime" : "2025-12-27T14:15:12.221GMT",
"endTimeEpoch" : 1766844912221,
+ "exitCode" : 0,
"lastUpdated" : "",
"lastUpdatedEpoch" : 0,
"sparkUser" : "<script>alert('XSS')</script>",
diff --git
a/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
index 06ef3a5ecc9f..fef54a1faa3f 100644
---
a/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
+++
b/core/src/test/resources/HistoryServerExpectations/limit_app_list_json_expectation.json
@@ -1,4 +1,22 @@
[ {
+ "attempts" : [ {
+ "appSparkVersion" : "3.5.2",
+ "completed" : true,
+ "duration" : 221145,
+ "endTime" : "2026-06-23T17:34:35.148GMT",
+ "endTimeEpoch" : 1782236075148,
+ "exitCode" : 1,
+ "logSourceFullPath" : "",
+ "logSourceName" : "",
+ "lastUpdated" : "",
+ "lastUpdatedEpoch" : 0,
+ "sparkUser" : "spark",
+ "startTime" : "2026-06-23T17:30:54.003GMT",
+ "startTimeEpoch" : 1782235854003
+ } ],
+ "id" : "local-1430917381536",
+ "name" : "Spark shell"
+}, {
"attempts" : [ {
"appSparkVersion" : "4.2.0-SNAPSHOT",
"logSourceFullPath" : "",
@@ -7,6 +25,7 @@
"duration" : 2006,
"endTime" : "2025-12-27T14:15:12.221GMT",
"endTimeEpoch" : 1766844912221,
+ "exitCode" : 0,
"lastUpdated" : "",
"lastUpdatedEpoch" : 0,
"sparkUser" : "<script>alert('XSS')</script>",
@@ -32,21 +51,4 @@
} ],
"id" : "local-1642039451826",
"name" : "Spark shell"
-}, {
- "attempts" : [ {
- "appSparkVersion" : "3.1.1.119",
- "logSourceFullPath" : "",
- "logSourceName" : "",
- "completed" : true,
- "duration" : 274875,
- "endTime" : "2021-08-10T23:29:30.208GMT",
- "endTimeEpoch" : 1628638170208,
- "lastUpdated" : "",
- "lastUpdatedEpoch" : 0,
- "sparkUser" : "john",
- "startTime" : "2021-08-10T23:24:55.333GMT",
- "startTimeEpoch" : 1628637895333
- } ],
- "id" : "application_1628109047826_1317105",
- "name" : "Spark shell"
} ]
diff --git
a/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
index a1d2f9e9ef4d..885045f2faf5 100644
---
a/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
+++
b/core/src/test/resources/HistoryServerExpectations/minDate_app_list_json_expectation.json
@@ -1,4 +1,22 @@
[ {
+ "attempts" : [ {
+ "appSparkVersion" : "3.5.2",
+ "completed" : true,
+ "duration" : 221145,
+ "endTime" : "2026-06-23T17:34:35.148GMT",
+ "endTimeEpoch" : 1782236075148,
+ "exitCode" : 1,
+ "logSourceFullPath" : "",
+ "logSourceName" : "",
+ "lastUpdated" : "",
+ "lastUpdatedEpoch" : 0,
+ "sparkUser" : "spark",
+ "startTime" : "2026-06-23T17:30:54.003GMT",
+ "startTimeEpoch" : 1782235854003
+ } ],
+ "id" : "local-1430917381536",
+ "name" : "Spark shell"
+}, {
"attempts" : [ {
"appSparkVersion" : "4.2.0-SNAPSHOT",
"logSourceFullPath" : "",
@@ -7,6 +25,7 @@
"duration" : 2006,
"endTime" : "2025-12-27T14:15:12.221GMT",
"endTimeEpoch" : 1766844912221,
+ "exitCode" : 0,
"lastUpdated" : "",
"lastUpdatedEpoch" : 0,
"sparkUser" : "<script>alert('XSS')</script>",
diff --git
a/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
index 413660c4b689..ead957a5fbbb 100644
---
a/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
+++
b/core/src/test/resources/HistoryServerExpectations/minEndDate_app_list_json_expectation.json
@@ -1,4 +1,22 @@
[ {
+ "attempts" : [ {
+ "appSparkVersion" : "3.5.2",
+ "completed" : true,
+ "duration" : 221145,
+ "endTime" : "2026-06-23T17:34:35.148GMT",
+ "endTimeEpoch" : 1782236075148,
+ "exitCode" : 1,
+ "logSourceFullPath" : "",
+ "logSourceName" : "",
+ "lastUpdated" : "",
+ "lastUpdatedEpoch" : 0,
+ "sparkUser" : "spark",
+ "startTime" : "2026-06-23T17:30:54.003GMT",
+ "startTimeEpoch" : 1782235854003
+ } ],
+ "id" : "local-1430917381536",
+ "name" : "Spark shell"
+}, {
"attempts" : [ {
"appSparkVersion" : "4.2.0-SNAPSHOT",
"logSourceFullPath" : "",
@@ -7,6 +25,7 @@
"duration" : 2006,
"endTime" : "2025-12-27T14:15:12.221GMT",
"endTimeEpoch" : 1766844912221,
+ "exitCode" : 0,
"lastUpdated" : "",
"lastUpdatedEpoch" : 0,
"sparkUser" : "<script>alert('XSS')</script>",
diff --git
a/core/src/test/resources/HistoryServerExpectations/one_app_with_non_zero_exit_code_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/one_app_with_non_zero_exit_code_json_expectation.json
new file mode 100644
index 000000000000..9b9ee4c7ed9d
--- /dev/null
+++
b/core/src/test/resources/HistoryServerExpectations/one_app_with_non_zero_exit_code_json_expectation.json
@@ -0,0 +1,19 @@
+{
+ "attempts" : [ {
+ "appSparkVersion" : "3.5.2",
+ "completed" : true,
+ "duration" : 221145,
+ "endTime" : "2026-06-23T17:34:35.148GMT",
+ "endTimeEpoch" : 1782236075148,
+ "exitCode" : 1,
+ "logSourceFullPath" : "",
+ "logSourceName" : "",
+ "lastUpdated" : "",
+ "lastUpdatedEpoch" : 0,
+ "sparkUser" : "spark",
+ "startTime" : "2026-06-23T17:30:54.003GMT",
+ "startTimeEpoch" : 1782235854003
+ } ],
+ "id" : "local-1430917381536",
+ "name" : "Spark shell"
+}
diff --git a/core/src/test/resources/spark-events/local-1430917381536
b/core/src/test/resources/spark-events/local-1430917381536
new file mode 100644
index 000000000000..c356d307895e
--- /dev/null
+++ b/core/src/test/resources/spark-events/local-1430917381536
@@ -0,0 +1,53 @@
+{"Event":"SparkListenerLogStart","Spark Version":"3.5.2"}
+{"Event":"SparkListenerResourceProfileAdded","Resource Profile Id":0,"Executor
Resource Requests":{"cores":{"Resource Name":"cores","Amount":1,"Discovery
Script":"","Vendor":""},"memory":{"Resource
Name":"memory","Amount":2048,"Discovery
Script":"","Vendor":""},"offHeap":{"Resource
Name":"offHeap","Amount":0,"Discovery Script":"","Vendor":""}},"Task Resource
Requests":{"cpus":{"Resource Name":"cpus","Amount":1.0}}}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":587552240,"JVMOffHeapMemory":169149216,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":41075,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9117978624,"ProcessTreeJVMRSSMemory":1213
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":587552240,"JVMOffHeapMemory":169436864,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":41075,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9117978624,"ProcessTreeJVMRSSMemory":1213
[...]
+{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor
ID":"driver","Host":"localhost","Port":24790},"Maximum
Memory":3676726886,"Timestamp":1782235880552,"Maximum Onheap
Memory":3676726886,"Maximum Offheap Memory":0}
+{"Event":"SparkListenerEnvironmentUpdate","JVM Information":{"Java
Home":"/usr/lib/jvm/java","Java Version":"17.0.5 (Microsoft)","Scala
Version":"version 2.12.18"},"Spark Properties":{},"Hadoop
Properties":{},"System Properties":{},"Metrics Properties":{},"Classpath
Entries":{}}
+{"Event":"SparkListenerApplicationStart","App Name":"Spark shell","App
ID":"local-1430917381536","Timestamp":1782235854003,"User":"spark"}
+{"Event":"org.apache.spark.scheduler.SparkListenerMiscellaneousProcessAdded","time":1782235881468,"processId":"yarn-am","info":{"hostPort":"localhost:8042","cores":1,"logUrlInfo":{"stdout":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000001/spark/stdout?start=-4096","stderr":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000001/spark/stderr?start=-4096"}}}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":663049712,"JVMOffHeapMemory":181725520,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":4311030,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9135951872,"ProcessTreeJVMRSSMemory":12
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":663049712,"JVMOffHeapMemory":182043856,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":4311030,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9135951872,"ProcessTreeJVMRSSMemory":12
[...]
+{"Event":"SparkListenerExecutorAdded","Timestamp":1782235902754,"Executor
ID":"1","Executor Info":{"Host":"localhost","Total Cores":1,"Log
Urls":{"stdout":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000002/spark/stdout?start=-4096","stderr":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000002/spark/stderr?start=-4096"},"Attributes":{"NM_HTTP_ADDRESS":"localhost:8042","USER":"spark","LOG_FILES":"stderr,stdout","NM_HTTP_PORT":"8042"
[...]
+{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor
ID":"1","Host":"localhost","Port":11366},"Maximum
Memory":1099746508,"Timestamp":1782235902912,"Maximum Onheap
Memory":1099746508,"Maximum Offheap Memory":0}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":684021232,"JVMOffHeapMemory":183676472,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":12723107,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9146445824,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorAdded","Timestamp":1782235908831,"Executor
ID":"3","Executor Info":{"Host":"localhost","Total Cores":1,"Log
Urls":{"stdout":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000003/spark/stdout?start=-4096","stderr":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000003/spark/stderr?start=-4096"},"Attributes":{"NM_HTTP_ADDRESS":"localhost:8042","USER":"spark","LOG_FILES":"stderr,stdout","NM_HTTP_PORT":"8042"
[...]
+{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor
ID":"3","Host":"localhost","Port":11298},"Maximum
Memory":1099746508,"Timestamp":1782235908994,"Maximum Onheap
Memory":1099746508,"Maximum Offheap Memory":0}
+{"Event":"SparkListenerExecutorAdded","Timestamp":1782235910397,"Executor
ID":"2","Executor Info":{"Host":"localhost","Total Cores":1,"Log
Urls":{"stdout":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000004/spark/stdout?start=-4096","stderr":"http://localhost:8042/node/containerlogs/container_e121_1430917381536_01_000004/spark/stderr?start=-4096"},"Attributes":{"NM_HTTP_ADDRESS":"localhost:8042","USER":"spark","LOG_FILES":"stderr,stdout","NM_HTTP_PORT":"8042"
[...]
+{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor
ID":"2","Host":"localhost","Port":16966},"Maximum
Memory":1099746508,"Timestamp":1782235910574,"Maximum Onheap
Memory":1099746508,"Maximum Offheap Memory":0}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":500075000,"JVMOffHeapMemory":200249712,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29547261,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9211654144,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":562989560,"JVMOffHeapMemory":207438768,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29659885,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9231601664,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":528762488,"JVMOffHeapMemory":230684216,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29659885,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9231601664,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":532956792,"JVMOffHeapMemory":230282752,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29668077,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9231601664,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":537151096,"JVMOffHeapMemory":232446888,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29668077,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9239990272,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":709117560,"JVMOffHeapMemory":244835032,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29816850,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9239990272,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":713311864,"JVMOffHeapMemory":244563024,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29816850,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9240129536,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorRemoved","Timestamp":1782235983810,"Executor
ID":"1","Removed Reason":"Executor decommission: spark scale down"}
+{"Event":"SparkListenerBlockManagerRemoved","Block Manager ID":{"Executor
ID":"1","Host":"localhost","Port":11366},"Timestamp":1782235983814}
+{"Event":"SparkListenerExecutorRemoved","Timestamp":1782235985799,"Executor
ID":"1","Removed Reason":"Executor for container
container_e121_1430917381536_01_000002 exited because of a YARN event (e.g.,
preemption) and not because of an error in the running job."}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":793003640,"JVMOffHeapMemory":248284608,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29808921,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorRemoved","Timestamp":1782235989914,"Executor
ID":"3","Removed Reason":"Executor decommission: spark scale down"}
+{"Event":"SparkListenerBlockManagerRemoved","Block Manager ID":{"Executor
ID":"3","Host":"localhost","Port":11298},"Timestamp":1782235989915}
+{"Event":"SparkListenerExecutorRemoved","Timestamp":1782235991801,"Executor
ID":"3","Removed Reason":"Executor for container
container_e121_1430917381536_01_000003 exited because of a YARN event (e.g.,
preemption) and not because of an error in the running job."}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":801392248,"JVMOffHeapMemory":249805880,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29808922,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":805586552,"JVMOffHeapMemory":249421632,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29800730,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":805586552,"JVMOffHeapMemory":249945392,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29800730,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":809780856,"JVMOffHeapMemory":250125488,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":29780056,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9248518144,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerJobStart","Job ID":0,"Submission
Time":1782236034538,"Stage Infos":[{"Stage ID":0,"Stage Attempt ID":0,"Stage
Name":"writeDataFrame at <console>:84","Number of Tasks":3,"RDD Info":[{"RDD
ID":3,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"0\",\"name\":\"WriteFiles\"}","Callsite":"writeDataFrame
at <console>:84","Parent IDs":[2],"Storage Level":{"Use Disk":false,"Use
Memory":false,"Use Off
Heap":false,"Deserialized":false,"Replication":1},"Barrier":false,"Determinist
[...]
+{"Event":"SparkListenerStageSubmitted","Stage Info":{"Stage ID":0,"Stage
Attempt ID":0,"Stage Name":"writeDataFrame at <console>:84","Number of
Tasks":3,"RDD Info":[{"RDD
ID":3,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"0\",\"name\":\"WriteFiles\"}","Callsite":"writeDataFrame
at <console>:84","Parent IDs":[2],"Storage Level":{"Use Disk":false,"Use
Memory":false,"Use Off
Heap":false,"Deserialized":false,"Replication":1},"Barrier":false,"DeterministicLevel":"DETERMINATE","Number
of Parti [...]
+{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task
Info":{"Task ID":0,"Index":0,"Attempt":0,"Partition ID":0,"Launch
Time":1782236034680,"Executor
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Resource Profile
ID":0,"Accumulables":[]}}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":673064904,"JVMOffHeapMemory":281199496,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33969867,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9349148672,"ProcessTreeJVMRS
[...]
+{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task
Info":{"Task ID":1,"Index":1,"Attempt":0,"Partition ID":1,"Launch
Time":1782236037542,"Executor
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Resource Profile
ID":0,"Accumulables":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task
ID":0,"Index":0,"Attempt":0,"Partition ID":0,"Launch
Time":1782236034680,"Executor
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
Result Time":0,"Finish
Time":1782236037555,"Failed":false,"Killed":false,"Resource Profile
ID":0,"Accumulables":[{"ID":4,"Name":"task commit
time","Update":"50","Value":"50","In [...]
+{"Event":"SparkListenerTaskStart","Stage ID":0,"Stage Attempt ID":0,"Task
Info":{"Task ID":2,"Index":2,"Attempt":0,"Partition ID":2,"Launch
Time":1782236037693,"Executor
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Resource Profile
ID":0,"Accumulables":[]}}
+{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task
ID":1,"Index":1,"Attempt":0,"Partition ID":1,"Launch
Time":1782236037542,"Executor
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
Result Time":0,"Finish
Time":1782236037694,"Failed":false,"Killed":false,"Resource Profile
ID":0,"Accumulables":[{"ID":4,"Name":"task commit
time","Update":"37","Value":"87","In [...]
+{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task
Type":"ResultTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task
ID":2,"Index":2,"Attempt":0,"Partition ID":2,"Launch
Time":1782236037693,"Executor
ID":"2","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting
Result Time":0,"Finish
Time":1782236037801,"Failed":false,"Killed":false,"Resource Profile
ID":0,"Accumulables":[{"ID":4,"Name":"task commit
time","Update":"40","Value":"127","I [...]
+{"Event":"SparkListenerStageExecutorMetrics","Executor ID":"2","Stage
ID":0,"Stage Attempt ID":0,"Executor
Metrics":{"JVMHeapMemory":186850048,"JVMOffHeapMemory":116149944,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":16807195,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":4315377664,"ProcessTreeJVMRSSMemory":628396032,"ProcessTreePythonVMemory":0,"ProcessTr
[...]
+{"Event":"SparkListenerStageExecutorMetrics","Executor ID":"driver","Stage
ID":0,"Stage Attempt ID":0,"Executor
Metrics":{"JVMHeapMemory":673064904,"JVMOffHeapMemory":281199496,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33969867,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9349148672,"ProcessTreeJVMRSSMemory":1489293312,"ProcessTreePythonVMemory":0,"Pro
[...]
+{"Event":"SparkListenerStageCompleted","Stage Info":{"Stage ID":0,"Stage
Attempt ID":0,"Stage Name":"writeDataFrame at <console>:84","Number of
Tasks":3,"RDD Info":[{"RDD
ID":3,"Name":"MapPartitionsRDD","Scope":"{\"id\":\"0\",\"name\":\"WriteFiles\"}","Callsite":"writeDataFrame
at <console>:84","Parent IDs":[2],"Storage Level":{"Use Disk":false,"Use
Memory":false,"Use Off
Heap":false,"Deserialized":false,"Replication":1},"Barrier":false,"DeterministicLevel":"DETERMINATE","Number
of Parti [...]
+{"Event":"SparkListenerJobEnd","Job ID":0,"Completion Time":1782236037805,"Job
Result":{"Result":"JobSucceeded"}}
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":798894024,"JVMOffHeapMemory":302532712,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33973602,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9373675520,"ProcessTreeJVMRS
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":807282632,"JVMOffHeapMemory":304663000,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33991159,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9373675520,"ProcessTreeJVMRS
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":811476936,"JVMOffHeapMemory":306466816,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":733937,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":733937,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33999351,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9373675520,"ProcessTreeJVMRS
[...]
+{"Event":"SparkListenerExecutorMetricsUpdate","Executor ID":"driver","Metrics
Updated":[],"Executor Metrics Updated":[{"Stage ID":-1,"Stage Attempt
ID":-1,"Executor
Metrics":{"JVMHeapMemory":542426624,"JVMOffHeapMemory":305486288,"OnHeapExecutionMemory":0,"OffHeapExecutionMemory":0,"OnHeapStorageMemory":0,"OffHeapStorageMemory":0,"OnHeapUnifiedMemory":0,"OffHeapUnifiedMemory":0,"DirectPoolMemory":33999351,"MappedPoolMemory":0,"ProcessTreeJVMVMemory":9391607808,"ProcessTreeJVMRSSMemory":1
[...]
+{"Event":"SparkListenerApplicationEnd","Timestamp":1782236075148,"ExitCode":1}
diff --git
a/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
index 8a4fd0ded959..01a77b880ebc 100644
---
a/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
+++
b/core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
@@ -122,7 +122,7 @@ abstract class FsHistoryProviderSuite extends SparkFunSuite
with Matchers with P
writeFile(newAppComplete, None,
SparkListenerApplicationStart(newAppComplete.getName(),
Some("new-app-complete"), 1L, "test",
None),
- SparkListenerApplicationEnd(5L)
+ SparkListenerApplicationEnd(5L, Some(1))
)
// Write a new-style application log.
@@ -154,7 +154,8 @@ abstract class FsHistoryProviderSuite extends SparkFunSuite
with Matchers with P
end: Long,
lastMod: Long,
user: String,
- completed: Boolean): ApplicationInfo = {
+ completed: Boolean,
+ exitCode: Option[Int] = None): ApplicationInfo = {
val duration = if (end > 0) end - start else 0
// Get log dir info from the provider
@@ -162,13 +163,13 @@ abstract class FsHistoryProviderSuite extends
SparkFunSuite with Matchers with P
val (logSourceName, logSourceFullPath) =
provider.getLogDirInfo(logPath)
new ApplicationInfo(id, name, None, None, None, None,
List(ApplicationAttemptInfo(None, new Date(start),
- new Date(end), new Date(lastMod), duration, user, completed,
SPARK_VERSION,
+ new Date(end), new Date(lastMod), duration, user, completed,
SPARK_VERSION, exitCode,
Some(logSourceName), Some(logSourceFullPath))))
}
// For completed files, lastUpdated would be lastModified time.
list(0) should be (makeAppInfo("new-app-complete",
newAppComplete.getName(), 1L, 5L,
- newAppComplete.lastModified(), "test", true))
+ newAppComplete.lastModified(), "test", true, Some(1)))
list(1) should be (makeAppInfo("new-complete-lzf",
newAppCompressedComplete.getName(),
1L, 4L, newAppCompressedComplete.lastModified(), "test", true))
diff --git
a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
index 13432b6ed9fc..79e0001fa22f 100644
---
a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
+++
b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
@@ -140,6 +140,7 @@ abstract class HistoryServerSuite extends SparkFunSuite
with BeforeAndAfter with
"limit app list json" -> "applications?limit=3",
"one app json" -> "applications/local-1422981780767",
"one app multi-attempt json" -> "applications/local-1426533911241",
+ "one app with non-zero exit code json" ->
"applications/local-1430917381536",
"job list json" -> "applications/local-1422981780767/jobs",
"job list from multi-attempt app json(1)" ->
"applications/local-1426533911241/1/jobs",
"job list from multi-attempt app json(2)" ->
"applications/local-1426533911241/2/jobs",
diff --git
a/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
b/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
index 193c2a9f3349..e0a6120b6a59 100644
--- a/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
@@ -200,6 +200,7 @@ abstract class AppStatusListenerSuite extends SparkFunSuite
with BeforeAndAfter
assert(attempt.endTime.getTime() === -1L)
assert(attempt.sparkUser === "user")
assert(!attempt.completed)
+ assert(attempt.exitCode === None)
assert(attempt.appSparkVersion === "TestSparkVersion")
}
@@ -757,7 +758,7 @@ abstract class AppStatusListenerSuite extends SparkFunSuite
with BeforeAndAfter
}
// End the application.
- listener.onApplicationEnd(SparkListenerApplicationEnd(42L))
+ listener.onApplicationEnd(SparkListenerApplicationEnd(42L, Some(1)))
check[ApplicationInfoWrapper]("id") { app =>
assert(app.info.name === "name")
@@ -772,6 +773,7 @@ abstract class AppStatusListenerSuite extends SparkFunSuite
with BeforeAndAfter
assert(attempt.duration === 41L)
assert(attempt.sparkUser === "user")
assert(attempt.completed)
+ assert(attempt.exitCode === Some(1))
}
}
diff --git
a/core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala
b/core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala
index 7bb36a97163b..23cb99bce2ff 100644
---
a/core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala
+++
b/core/src/test/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializerSuite.scala
@@ -399,7 +399,8 @@ class KVStoreProtobufSerializerSuite extends SparkFunSuite {
duration = 100,
sparkUser = null,
completed = true,
- appSparkVersion = null
+ appSparkVersion = null,
+ exitCode = Some(1)
))
val input = new ApplicationInfoWrapper(
ApplicationInfo(
diff --git a/dev/.rat-excludes b/dev/.rat-excludes
index 2eca24afcec5..4aee210e85b9 100644
--- a/dev/.rat-excludes
+++ b/dev/.rat-excludes
@@ -84,6 +84,7 @@ local-1426633911242
local-1430917381534
local-1430917381535_1
local-1430917381535_2
+local-1430917381536
local-1642039451826
eventlog_v2_local-1766844910796
DESCRIPTION
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index 984ce2f37ce5..cfd64334ae91 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -59,6 +59,9 @@ object MimaExcludes {
// Exclude rules for 4.2.x from 4.1.0
lazy val v42excludes = v41excludes ++ Seq(
+ // [SPARK-54879] Add exitCode field to ApplicationAttemptInfo
+
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.ApplicationAttemptInfo.tupled"),
+
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.ApplicationAttemptInfo.curried"),
// [SQL] SafeJsonSerializer.safeMapToJValue: second parameter widened from
Function1 to
// Function2 so the key is passed to the value serializer
(progress.scala). Binary-incompatible
// vs spark-sql-api 4.0.0; not part of the public supported API
(private[streaming] package).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]