Thejdeep created SPARK-43334:
--------------------------------
Summary: Error while serializing ExecutorPeakMetricsDistributions
into API JSON response
Key: SPARK-43334
URL: https://issues.apache.org/jira/browse/SPARK-43334
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 3.3.3
Reporter: Thejdeep
When we try to get the ExecutorPeakMetricsDistributions [through the
API|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/status/api/v1/api.scala#L463]
(/stages), there is a possibility of encountering an issue while serializing
the StagesData into a JSON if the executor metrics are empty.
The following error is thrown :
{code:java}
Caused by: com.fasterxml.jackson.databind.JsonMappingException: -1 (through
reference chain: scala.c
ollection.immutable.$colon$colon[0]->org.apache.spark.status.api.v1.StageData["executorMetricsDistri
butions"]->org.apache.spark.status.api.v1.ExecutorMetricsDistributions["peakMemoryMetrics"])
at
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.jav
a:390)
at
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.jav
a:349) {code}
This happens because the indices for the quartiles are populated incorrectly as
-1 since the metrics itself are empty and this leads to this exception being
thrown.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
