If I understood this correctly, then I think you can also think of this from the opposite direction. Instead of moving sources/sinks OUT of Flume, let them develop outside Flume, and if they prove to be good, in high demand, often used with Flume, or in need of "adoption" by core Flume developers, then BRING IN such sources/sinks and, optionally(?), their authors with them. Maybe this way you can have the best of both worlds.
If I develop source/sink for the new, fancy, FooBarBaz data store, can I get it in Flume? No. and that's good, because if FooBarBaz doesn't take off, who'll want to maintain it? If I develop source/sink for Kafka, which is known to be very popular and often used with Flume, and there are either core Flume developers interested in maintaining this source/sink, or if there are external developers who have been working on this outside Apache (i.e. the chances are they'll follow the project to Apache and continue contributing to it, eventually earning the committer rights), then this source/sink for Kafka should be considered for adoption by Flume. This way you let the external ecosystem live on, develop independently, compete for user and developer adoption, and once good sources/sinks emerge from there, they can be moved under Flume if everyone involved agrees that's the right thing to do. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Fri, Nov 29, 2013 at 10:11 AM, Frank Yao <[email protected]> wrote: > I'm the 'guys' who is working on Kafka source/sinks:). > > A feature for popular, fast-growing and mature products is necessary to > merge into Flume. Why? > a) sources/sinks of mature products are really a motivation to flume users. > b) developers have willing to add new sources/sinks to flume. > > For a), early there is a developer said, ' Having to shop around for > various sources/sinks is more troublesome since I have to first find which > flavor of a given sink is being maintained today, deal with licenses, > incompatibilities, mismatch versions, upgrades, deployment, not fixed bugs > and wondering if this is even going to work at all.' I thinks this is > common for people who want to use flume but cannot found what he wanted at > first. > For b), if developers give plugins to flume and are rejected only because > of keeping Flume lean, developers will lose their passion of contributing > to Flume. > > I think if developers made sources/sinks for Flume, and he/she thought the > sources/sinks were in great need by Flume users, he/she need to request a > vote by committers, if most committers think it's necessary, then merge > these to Flume. If not, then move these to Flume-contrib projects like > what ElasticSearch does. > > > > ----------------- > > 姚仁捷 Frank Yao > @超大杯摩卡星冰乐 <http://weibo.com/frankymryao> > http://baniu.me > Vipshop, Shanghai > > > > > On Fri, Nov 29, 2013 at 3:17 PM, Otis Gospodnetic < > [email protected]> wrote: > > > If you think "ecosystem", and an OSS project like Flume should very much > > think ecosystem, then leaving things on Github, etc. probably makes more > > sense. Over the years (now over a decade!) I've witnessed what happens > > with contrib/-type approach - authors need to have access to maintain > their > > stuff, make it work with the build system changes, make it work before > > released, etc. etc,, which is all hard, and very often you can't just > give > > contrib authors Apache commit rights. So instead of trying to pull > > everything in, one should focus on *making developer-friendly core/APIs". > > Developers will then build tools that work with this core and naturally > > create a rich ecosystem of tools around it. > > > > Otis > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > On Nov 29, 2013 12:14 AM, "Jeremy Karlson" <[email protected]> > > wrote: > > > > > As someone who just developed a sink, let me add my two cents. > > > > > > If the intention is to separate “core Flume” from second class citizens > > > like myself ( ;-) ), a contrib module only makes sense if those > > > contributors can manage fixes and commit to their modules themselves. > > > Waiting for core developers to apply changes to modules they don’t > want > > to > > > work on will just leave maintainers like myself annoyed at waiting and > > core > > > contributors annoyed at having to do it. I think you’d have to hand > out > > > commit abilities to several people for there to be smiles all round. > > > > > > If you don’t want to or can’t do that (understandable), maybe just let > > > everyone do their own module management on GitHub or whatever, and > > provide > > > a page that links to “add on” modules. (This is the approach > > Elasticsearch > > > takes, I think.) > > > > > > -- Jeremy > > > > > > > > > On Nov 28, 2013, at 11:06, Hari Shreedharan <[email protected] > > > > > wrote: > > > > > > > Juhani and others, > > > > > > > > I agree that it does make sense to add a contrib module to flume > where > > > > non-hadoopy stuff can go. I will start a discussion on this early > next > > > week. > > > > > > > > Hari > > > > > > > > On Thursday, November 28, 2013, Steve Morin wrote: > > > > > > > >> Israel, > > > >> I guess my questions is why the suggestion to use the elastic search > > > >> model, is there something you see that's not working? > > > >> -Steve > > > >> > > > >> > > > >> On Mon, Nov 25, 2013 at 5:34 PM, Israel Ekpo <[email protected] > > > <javascript:;>> > > > >> wrote: > > > >> > > > >>> I think we can take a page of out the ElasticSearch playbook. > > > >>> > > > >>> > > > >>> > > > >> > > > > > > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html > > > >>> > > > >>> I like the model they follow. > > > >>> > > > >>> The Flume architecture makes it easy for plugins at any layer > > (source, > > > >>> interceptor, sink etc) > > > >>> > > > >>> Contributors can host plugins on github and manage the > documentation > > > and > > > >>> maintenance of the plugin. > > > >>> > > > >>> Others can chip it when possible to improve or maintain the > plugins. > > > >>> > > > >>> This will still allow new features to the project without > necessarily > > > >>> meaning that Flume committers are on the hook for maintaining it. > > > >>> > > > >>> > > > >>> > > > >>> *Author and Instructor for the Upcoming Book and Lecture Series* > > > >>> *Massive Log Data Aggregation, Processing, Searching and > > Visualization > > > >> with > > > >>> Open Source Software* > > > >>> *http://massivelogdata.com <http://massivelogdata.com>* > > > >>> > > > >>> > > > >>> On Mon, Nov 25, 2013 at 3:14 AM, Juhani Connolly < > > > >>> [email protected] <javascript:;>> wrote: > > > >>> > > > >>>> Hey guys, > > > >>>> > > > >>>> What I write here is all just my personal opinion and I'm writing > in > > > >>> hopes > > > >>>> of starting a discussion and/or getting feedback. I know I've not > > been > > > >>> very > > > >>>> active on the project recently(due to other engagements) but do > > still > > > >>> want > > > >>>> it to succeed and hope to find more time for it eventually. > > > >>>> > > > >>>> Right now I see new/active issues for the addition of Redis and > > Kafka > > > >>>> sinks, and while they're nice features, I'm personally concerned > > about > > > >>>> feature bloat of the project. There are dozens of interceptors, > > sinks > > > >> and > > > >>>> sources that can be thought of, but most of them are very specific > > to > > > a > > > >>>> specific use-case. > > > >>>> > > > >>>> Every time we add a new component we're also committing to > > maintaining > > > >> it > > > >>>> over future releases, even if the original contributor gets too > busy > > > >> for > > > >>>> it. The more such components get added, the more we will get > > > distracted > > > >>>> from improving core features and getting rid of issues affecting > > them. > > > >>>> > > > >>>> For these reasons I generally haven't submitted components we > > > developed > > > >>>> for internal use(because they are too specific to our use cases), > > just > > > >>>> passing back fixes that fix bugs or apply to the core project. > > > >>>> > > > >>>> For these reasons I think we may want to consider either a) being > > more > > > >>>> selective regarding additional component submissions or b) make a > > > >> contrib > > > >>>> directory to the project which includes the components but doesn't > > > >>>> guarrantee ongoing maintenance or compatibility. > > > >>>> > > > >>>> On the flip side of course, taking approaches like this may > > discourage > > > >>> new > > > >>>> contributors and could thus be considered a negative, and if many > > > >> people > > > >>>> feel this way they should definitely share their thoughts. > > > >>>> > > > >>>> I'd be curious to know what others think, and what direction they > > hope > > > >> to > > > >>>> take the project in the future. > > > >>>> > > > >>> > > > >> > > > > > > > > >
