I'm the 'guys' who is working on Kafka source/sinks:). A feature for popular, fast-growing and mature products is necessary to merge into Flume. Why? a) sources/sinks of mature products are really a motivation to flume users. b) developers have willing to add new sources/sinks to flume.
For a), early there is a developer said, ' Having to shop around for various sources/sinks is more troublesome since I have to first find which flavor of a given sink is being maintained today, deal with licenses, incompatibilities, mismatch versions, upgrades, deployment, not fixed bugs and wondering if this is even going to work at all.' I thinks this is common for people who want to use flume but cannot found what he wanted at first. For b), if developers give plugins to flume and are rejected only because of keeping Flume lean, developers will lose their passion of contributing to Flume. I think if developers made sources/sinks for Flume, and he/she thought the sources/sinks were in great need by Flume users, he/she need to request a vote by committers, if most committers think it's necessary, then merge these to Flume. If not, then move these to Flume-contrib projects like what ElasticSearch does. ----------------- 姚仁捷 Frank Yao @超大杯摩卡星冰乐 <http://weibo.com/frankymryao> http://baniu.me Vipshop, Shanghai On Fri, Nov 29, 2013 at 3:17 PM, Otis Gospodnetic < [email protected]> wrote: > If you think "ecosystem", and an OSS project like Flume should very much > think ecosystem, then leaving things on Github, etc. probably makes more > sense. Over the years (now over a decade!) I've witnessed what happens > with contrib/-type approach - authors need to have access to maintain their > stuff, make it work with the build system changes, make it work before > released, etc. etc,, which is all hard, and very often you can't just give > contrib authors Apache commit rights. So instead of trying to pull > everything in, one should focus on *making developer-friendly core/APIs". > Developers will then build tools that work with this core and naturally > create a rich ecosystem of tools around it. > > Otis > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > > > > On Nov 29, 2013 12:14 AM, "Jeremy Karlson" <[email protected]> > wrote: > > > As someone who just developed a sink, let me add my two cents. > > > > If the intention is to separate “core Flume” from second class citizens > > like myself ( ;-) ), a contrib module only makes sense if those > > contributors can manage fixes and commit to their modules themselves. > > Waiting for core developers to apply changes to modules they don’t want > to > > work on will just leave maintainers like myself annoyed at waiting and > core > > contributors annoyed at having to do it. I think you’d have to hand out > > commit abilities to several people for there to be smiles all round. > > > > If you don’t want to or can’t do that (understandable), maybe just let > > everyone do their own module management on GitHub or whatever, and > provide > > a page that links to “add on” modules. (This is the approach > Elasticsearch > > takes, I think.) > > > > -- Jeremy > > > > > > On Nov 28, 2013, at 11:06, Hari Shreedharan <[email protected]> > > wrote: > > > > > Juhani and others, > > > > > > I agree that it does make sense to add a contrib module to flume where > > > non-hadoopy stuff can go. I will start a discussion on this early next > > week. > > > > > > Hari > > > > > > On Thursday, November 28, 2013, Steve Morin wrote: > > > > > >> Israel, > > >> I guess my questions is why the suggestion to use the elastic search > > >> model, is there something you see that's not working? > > >> -Steve > > >> > > >> > > >> On Mon, Nov 25, 2013 at 5:34 PM, Israel Ekpo <[email protected] > > <javascript:;>> > > >> wrote: > > >> > > >>> I think we can take a page of out the ElasticSearch playbook. > > >>> > > >>> > > >>> > > >> > > > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html > > >>> > > >>> I like the model they follow. > > >>> > > >>> The Flume architecture makes it easy for plugins at any layer > (source, > > >>> interceptor, sink etc) > > >>> > > >>> Contributors can host plugins on github and manage the documentation > > and > > >>> maintenance of the plugin. > > >>> > > >>> Others can chip it when possible to improve or maintain the plugins. > > >>> > > >>> This will still allow new features to the project without necessarily > > >>> meaning that Flume committers are on the hook for maintaining it. > > >>> > > >>> > > >>> > > >>> *Author and Instructor for the Upcoming Book and Lecture Series* > > >>> *Massive Log Data Aggregation, Processing, Searching and > Visualization > > >> with > > >>> Open Source Software* > > >>> *http://massivelogdata.com <http://massivelogdata.com>* > > >>> > > >>> > > >>> On Mon, Nov 25, 2013 at 3:14 AM, Juhani Connolly < > > >>> [email protected] <javascript:;>> wrote: > > >>> > > >>>> Hey guys, > > >>>> > > >>>> What I write here is all just my personal opinion and I'm writing in > > >>> hopes > > >>>> of starting a discussion and/or getting feedback. I know I've not > been > > >>> very > > >>>> active on the project recently(due to other engagements) but do > still > > >>> want > > >>>> it to succeed and hope to find more time for it eventually. > > >>>> > > >>>> Right now I see new/active issues for the addition of Redis and > Kafka > > >>>> sinks, and while they're nice features, I'm personally concerned > about > > >>>> feature bloat of the project. There are dozens of interceptors, > sinks > > >> and > > >>>> sources that can be thought of, but most of them are very specific > to > > a > > >>>> specific use-case. > > >>>> > > >>>> Every time we add a new component we're also committing to > maintaining > > >> it > > >>>> over future releases, even if the original contributor gets too busy > > >> for > > >>>> it. The more such components get added, the more we will get > > distracted > > >>>> from improving core features and getting rid of issues affecting > them. > > >>>> > > >>>> For these reasons I generally haven't submitted components we > > developed > > >>>> for internal use(because they are too specific to our use cases), > just > > >>>> passing back fixes that fix bugs or apply to the core project. > > >>>> > > >>>> For these reasons I think we may want to consider either a) being > more > > >>>> selective regarding additional component submissions or b) make a > > >> contrib > > >>>> directory to the project which includes the components but doesn't > > >>>> guarrantee ongoing maintenance or compatibility. > > >>>> > > >>>> On the flip side of course, taking approaches like this may > discourage > > >>> new > > >>>> contributors and could thus be considered a negative, and if many > > >> people > > >>>> feel this way they should definitely share their thoughts. > > >>>> > > >>>> I'd be curious to know what others think, and what direction they > hope > > >> to > > >>>> take the project in the future. > > >>>> > > >>> > > >> > > > > >
