[
https://issues.apache.org/jira/browse/SPARK-21598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110028#comment-16110028
]
Marcelo Vanzin commented on SPARK-21598:
----------------------------------------
[~steve_l] has been working for some time on SPARK-11373, which has some
overlapping goals. I'm fine with adding metrics, I just want the metrics
exposed to make sense.
This seems to be a different way to expose those metrics, so I'd rather pick
one way of doing it. The approach in SPARK-11373 integrates with existing tools
that collect metrics, while this approach would require people to write custom
code, which makes me by default prefer the former.
> Collect usability/events information from Spark History Server
> --------------------------------------------------------------
>
> Key: SPARK-21598
> URL: https://issues.apache.org/jira/browse/SPARK-21598
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 2.0.2
> Reporter: Eric Vandenberg
> Priority: Minor
>
> The Spark History Server doesn't currently have a way to collect
> usability/performance on its main activity, loading/replay of history files.
> We'd like to collect this information to monitor, target and measure
> improvements in the spark debugging experience (via history server usage.)
> Once available these usability events could be analyzed using other analytics
> tools.
> The event info to collect:
> SparkHistoryReplayEvent(
> logPath: String,
> logCompressionType: String,
> logReplayException: String // if an error
> logReplayAction: String // user replay, vs checkForLogs replay
> logCompleteFlag: Boolean,
> logFileSize: Long,
> logFileSizeUncompressed: Long,
> logLastModifiedTimestamp: Long,
> logCreationTimestamp: Long,
> logJobId: Long,
> logNumEvents: Int,
> logNumStages: Int,
> logNumTasks: Int
> logReplayDurationMillis: Long
> )
> The main spark engine has a SparkListenerInterface through which all compute
> engine events are broadcast. It probably doesn't make sense to reuse this
> abstraction for broadcasting spark history server events since the "events"
> are not related or compatible with one another. Also note the metrics
> registry collects history caching metrics but doesn't provide the type of
> above information.
> Proposal here would be to add some basic event listener infrastructure to
> capture history server activity events. This would work similar to how the
> SparkListener infrastructure works. It could be configured in a similar
> manner, eg. spark.history.listeners=MyHistoryListenerClass.
> Open to feedback / suggestions / comments on the approach or alternatives.
> cc: [~vanzin] [~cloud_fan] [~ajbozarth] [~jiangxb1987]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
