wypoon commented on a change in pull request #23767: [SPARK-26329][CORE] Faster
polling of executor memory metrics.
URL: https://github.com/apache/spark/pull/23767#discussion_r269310707
##########
File path:
core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
##########
@@ -274,6 +276,93 @@ class EventLoggingListenerSuite extends SparkFunSuite
with LocalSparkContext wit
val eventLogger = new EventLoggingListener(logName, None,
testDirPath.toUri(), conf)
val listenerBus = new LiveListenerBus(conf)
+ // Executor metrics
+ // driver
+ val md_1 = Array(4000L, 50L, 0L, 0L, 40L, 0L, 40L, 0L, 70L, 0L, 7500L,
3500L,
+ 0L, 0L, 0L, 0L, 10L, 90L, 2L, 20L)
+ val md_2 = Array(4500L, 50L, 0L, 0L, 40L, 0L, 40L, 0L, 70L, 0L, 8000L,
3500L,
+ 0L, 0L, 0L, 0L, 10L, 90L, 3L, 20L)
+ val md_3 = Array(4200L, 50L, 0L, 0L, 40L, 0L, 40L, 0L, 70L, 0L, 7800L,
3500L,
+ 0L, 0L, 0L, 0L, 15L, 100L, 5L, 20L)
Review comment:
I agree that it is virtually impossible to reason about the arrays of
numbers.
However, that is why I introduced:
```
// calculated metric peaks per stage per executor
// metrics sent during stage 0 for each executor
val cp0_1 = Seq(m1_1, m1_2, m1_3, t1, m1_4, t3).reduceLeft(max)
val cp0_2 = Seq(m2_1, m2_2, m2_3, t2, m2_4, t4).reduceLeft(max)
val cp0_d = Seq(md_1, md_2).reduceLeft(max)
// metrics sent during stage 1 for each executor
val cp1_1 = Seq(m1_4, m1_5, m1_6, m1_7, t5).reduceLeft(max)
val cp1_2 = Seq(m2_4, m2_5, m2_6, m2_7, t6).reduceLeft(max)
val cp1_d = Seq(md_2, md_3).reduceLeft(max)
```
and the internal check that the calculated peaks match the peak values that
I check against at the end.
When you look at what events are sent, you can match them with the above
definitions, and you can reason about the correctness of the metric aggregation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
