Jacek Lewandowski created SPARK-7171:
----------------------------------------
Summary: Allow for more flexible use of metric sources
Key: SPARK-7171
URL: https://issues.apache.org/jira/browse/SPARK-7171
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.3.1, 1.4.0
Reporter: Jacek Lewandowski
With the current API, the user is allowed to add a custom metric source by
providing its class in metrics configuration. Metrics themselves are provided
by Codahale and therefore they allow to register multiple metrics in a single
source. Basically we can break the available types of metrics into two types:
"push" and "pull" - by push metrics I mean that some execution code updates the
metric by itself either periodically or every n events. On the other hand, the
pull metrics include some function which pulls the data from the execution
environment, when triggered.
h5.Problem
The metric source is instantiated and registered during initialisation. Then,
the user has no way to access the instantiated object. It is also almost
impossible to access the execution environment of the current task. Therefore,
the user who wanted to provide his own {{RDD}} implementation along with a
dedicated metrics source, would find it very difficult to do this in a safe,
concise and elegant way.
h5.Proposed solution
At least, for the "push" metrics, it would be nice to be able to retrieve the
metrics source of particular type or with particular id from {{TaskContext}}.
It would allow custom tasks to update various metrics and would greatly improve
the usability of metrics.
This could be achieved quite easily since {{TaskContext}} is created by
{{Executor}}, which has access to the metrics system, it would inject some
method to retrieve the particular metrics source.
This solution wouldn't change the current API, but introducing one more method
in {{TaskContext}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
