[ https://issues.apache.org/jira/browse/BEAM-579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432450#comment-15432450 ]
Stas Levin commented on BEAM-579: --------------------------------- I've started working on it and should have a patch pretty soon. > Integrate NamedAggregators into Spark's sink system > --------------------------------------------------- > > Key: BEAM-579 > URL: https://issues.apache.org/jira/browse/BEAM-579 > Project: Beam > Issue Type: Task > Components: runner-spark > Reporter: Stas Levin > Assignee: Amit Sela > Priority: Critical > > At the moment {{NamedAggregators}} is an adapter between Beam's > {{Aggregator}} and Spark's {{Accumulator}} and is implemented as a single > Spark {{Accumulator}}, holding a map of metrics that can be augmented with > new metrics dynamically, after the the pipeline has already started. > Spark's out-of-the-box metrics mechanism does not support adding metrics to > {{Source}} s that have already registered (it pulls their metrics upon > registration and never updates them again). > In light of the above, it would seem that there is a gap to bridge between > the dynamic nature of {{NamedAggregators}} and Spark's current metric system > so that metrics that are added dynamically are also reported to the defined > Spark {{Sink}} s. -- This message was sent by Atlassian JIRA (v6.3.4#6332)