[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30573: [SPARK-26341][CORE]Expose executor memory metrics at the stage level, in the Stages tab

[GitBox]

AngersZhuuuu commented on a change in pull request #30573:
URL: https://github.com/apache/spark/pull/30573#discussion_r535931995



##########
File path: core/src/main/scala/org/apache/spark/status/LiveEntity.scala
##########
@@ -386,7 +386,7 @@ private class LiveExecutorStageSummary(
       metrics.memoryBytesSpilled,
       metrics.diskBytesSpilled,
       isExcluded,
-      Some(peakExecutorMetrics).filter(_.isSet),
+      Some(peakExecutorMetrics),

Review comment:
       > Then we need to set the filed as non-optional in `StageData`, or 
change `ExecutorMetricsJsonDeserializer`
   
   If we change field as non-optional in `StageData`, we also need to change 
`ExecutorMetricsJsonDeserializer` too since it based on the field's type , and 
it was used in three places,
   
![image](https://user-images.githubusercontent.com/46485123/101142364-77cae000-3650-11eb-92b1-8e39531f36c6.png)
   Just change here and remove filter may be the safest way.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org