Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/21221#discussion_r203100978
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala ---
@@ -160,11 +160,29 @@ case class
SparkListenerBlockUpdated(blockUpdatedInfo: BlockUpdatedInfo) extends
* Periodic updates from executors.
* @param execId executor id
* @param accumUpdates sequence of (taskId, stageId, stageAttemptId,
accumUpdates)
+ * @param executorUpdates executor level metrics updates
*/
@DeveloperApi
case class SparkListenerExecutorMetricsUpdate(
execId: String,
- accumUpdates: Seq[(Long, Int, Int, Seq[AccumulableInfo])])
+ accumUpdates: Seq[(Long, Int, Int, Seq[AccumulableInfo])],
+ executorUpdates: Option[Array[Long]] = None)
+ extends SparkListenerEvent
+
+/**
+ * Peak metric values for the executor for the stage, written to the
history log at stage
+ * completion.
+ * @param execId executor id
+ * @param stageId stage id
+ * @param stageAttemptId stage attempt
+ * @param executorMetrics executor level metrics, indexed by
MetricGetter.values
+ */
+@DeveloperApi
+case class SparkListenerStageExecutorMetrics(
+ execId: String,
+ stageId: Int,
+ stageAttemptId: Int,
+ executorMetrics: Array[Long])
--- End diff --
sorry was out for a bit, catching up now --
so I think these are valid concerns, but I'd really like to have an
approach which avoids all the getFooBar() methods -- its more error prone
internally with all the places you've got to make sure you match things up
exactly, and even for external consumers its nice to have an easy way to
iterate over everything. I think we can satisfy all concerns by making the
field & constructor private and adding a `getMetricValue(metric:
MetricGetter)`. perhaps we should also rename `MetricGetter` -- externally its
more like `ExecutorMetricType`. Users shouldn't know anything about its
ability to grab the value of the metric in a running executor.
I'm not totally stuck on this, but I'd certainly appreciate it -- I'm very
partial to enums, but realize not everyone sees the beauty of them like I do :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
