+1 on exposing the source/sink interface, since MetricsSystem by natural support plugin source and sink, so supporting this don't require a big change of current code. Also there's lot of requirements to add custom sink and source to the MetricsSystem, it is not suitable to maintain these code in Spark codebase.
Previously we use some ways like SparkEnv.metricsSystem to register source and sink, I think it is quick hacky, so exposing these interface will be better for external developers to better use this MetricsSystem. I'm going to open a JIRA for this. Thanks Jerry On Tue, Mar 22, 2016 at 6:29 PM, Steve Loughran <ste...@hortonworks.com> wrote: > > On 19 Mar 2016, at 16:16, Pete Robbins <robbin...@gmail.com> wrote: > > > There are several open Jiras to add new Sinks > > OpenTSDB https://issues.apache.org/jira/browse/SPARK-12194 > StatsD https://issues.apache.org/jira/browse/SPARK-11574 > > > > statsd is nicely easy to test: either listen in on a (localhost, port) or > simply create a socket and force it into the sink for the test run > > > Kafka https://issues.apache.org/jira/browse/SPARK-13392 > > Some have PRs from 2015 so I'm assuming there is not the desire to > integrate these into core Spark. Opening up the Sink/Source interfaces > would at least allow these to exist somewhere such as spark-packages > without having to pollute the o.a.s namespace > > > On Sat, 19 Mar 2016 at 13:05 Gerard Maas <gerard.m...@gmail.com> wrote: > >> +1 >> On Mar 19, 2016 08:33, "Pete Robbins" <robbin...@gmail.com> wrote: >> >>> This seems to me to be unnecessarily restrictive. These are very useful >>> extension points for adding 3rd party sources and sinks. >>> >>> I intend to make an Elasticsearch sink available on spark-packages but >>> this will require a single class, the sink, to be in the org.apache.spark >>> package tree. I could submit the package as a PR to the Spark codebase, and >>> I'd be happy to do that but it could be a completely separate add-on. >>> >>> There are similar issues with writing a 3rd party metrics source which >>> may not be of interest to the community at large so would probably not >>> warrant inclusion in the Spark codebase. >>> >>> Any thoughts? >>> >> >
