[
https://issues.apache.org/jira/browse/SPARK-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14521130#comment-14521130
]
Apache Spark commented on SPARK-7171:
-------------------------------------
User 'jacek-lewandowski' has created a pull request for this issue:
https://github.com/apache/spark/pull/5805
> Allow for more flexible use of metric sources
> ---------------------------------------------
>
> Key: SPARK-7171
> URL: https://issues.apache.org/jira/browse/SPARK-7171
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.3.1
> Reporter: Jacek Lewandowski
> Priority: Minor
>
> With the current API, the user is allowed to add a custom metric source by
> providing its class in metrics configuration. Metrics themselves are provided
> by Codahale and therefore they allow to register multiple metrics in a single
> source. Basically we can break the available types of metrics into two types:
> "push" and "pull" - by push metrics I mean that some execution code updates
> the metric by itself either periodically or every n events. On the other
> hand, the pull metrics include some function which pulls the data from the
> execution environment, when triggered.
> h5.Problem
> The metric source is instantiated and registered during initialisation. Then,
> the user has no way to access the instantiated object. It is also almost
> impossible to access the execution environment of the current task.
> Therefore, the user who wanted to provide his own {{RDD}} implementation
> along with a dedicated metrics source, would find it very difficult to do
> this in a safe, concise and elegant way.
> h5.Proposed solution
> At least, for the "push" metrics, it would be nice to be able to retrieve the
> metrics source of particular type or with particular id from {{TaskContext}}.
> It would allow custom tasks to update various metrics and would greatly
> improve the usability of metrics.
> This could be achieved quite easily since {{TaskContext}} is created by
> {{Executor}}, which has access to the metrics system, it would inject some
> method to retrieve the particular metrics source.
> This solution wouldn't change the current API, but just introduce one more
> method in {{TaskContext}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
