Jean-Baptiste Onofré created BEAM-2456:
------------------------------------------
Summary: Introduce generic metric poll thread interceptor
Key: BEAM-2456
URL: https://issues.apache.org/jira/browse/BEAM-2456
Project: Beam
Issue Type: New Feature
Components: sdk-ideas
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré
The Spark runner provides a convenient feature which is the metric sink.
By configuration, it allows us to configure a metric sink using
{{metrics.properties}} configuration containing:
{code}
driver.sink.graphite.class=org.apache.beam.runners.spark.metrics.sink.GraphiteSink
driver.sink.graphite.host=localhost
driver.sink.graphite.port=2003
driver.sink.graphite.prefix=spark
driver.sink.graphite.period=1
driver.sink.graphite.unit=SECONDS
{code}
This approach is very convenient to send the metric to any sink. I think we can
apply this logic in generic way working with any runner.
The idea is to use a metric poll thread in the pipeline (that we can enable via
{{PipelineOptions}}) and send to a sink.
I started a PoC about that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
