Frank, I would definitely vote for Kafka source/sink inclusion into flume. -Steve
On Fri, Nov 29, 2013 at 7:11 AM, Frank Yao <[email protected]> wrote: > I'm the 'guys' who is working on Kafka source/sinks:). > > A feature for popular, fast-growing and mature products is necessary to > merge into Flume. Why? > a) sources/sinks of mature products are really a motivation to flume users. > b) developers have willing to add new sources/sinks to flume. > > For a), early there is a developer said, ' Having to shop around for > various sources/sinks is more troublesome since I have to first find which > flavor of a given sink is being maintained today, deal with licenses, > incompatibilities, mismatch versions, upgrades, deployment, not fixed bugs > and wondering if this is even going to work at all.' I thinks this is > common for people who want to use flume but cannot found what he wanted at > first. > For b), if developers give plugins to flume and are rejected only because > of keeping Flume lean, developers will lose their passion of contributing > to Flume. > > I think if developers made sources/sinks for Flume, and he/she thought the > sources/sinks were in great need by Flume users, he/she need to request a > vote by committers, if most committers think it's necessary, then merge > these to Flume. If not, then move these to Flume-contrib projects like > what ElasticSearch does. > > > > ----------------- > > 姚仁捷 Frank Yao > @超大杯摩卡星冰乐 <http://weibo.com/frankymryao> > http://baniu.me > Vipshop, Shanghai > > > > > On Fri, Nov 29, 2013 at 3:17 PM, Otis Gospodnetic < > [email protected]> wrote: > > > If you think "ecosystem", and an OSS project like Flume should very much > > think ecosystem, then leaving things on Github, etc. probably makes more > > sense. Over the years (now over a decade!) I've witnessed what happens > > with contrib/-type approach - authors need to have access to maintain > their > > stuff, make it work with the build system changes, make it work before > > released, etc. etc,, which is all hard, and very often you can't just > give > > contrib authors Apache commit rights. So instead of trying to pull > > everything in, one should focus on *making developer-friendly core/APIs". > > Developers will then build tools that work with this core and naturally > > create a rich ecosystem of tools around it. > > > > Otis > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > On Nov 29, 2013 12:14 AM, "Jeremy Karlson" <[email protected]> > > wrote: > > > > > As someone who just developed a sink, let me add my two cents. > > > > > > If the intention is to separate “core Flume” from second class citizens > > > like myself ( ;-) ), a contrib module only makes sense if those > > > contributors can manage fixes and commit to their modules themselves. > > > Waiting for core developers to apply changes to modules they don’t > want > > to > > > work on will just leave maintainers like myself annoyed at waiting and > > core > > > contributors annoyed at having to do it. I think you’d have to hand > out > > > commit abilities to several people for there to be smiles all round. > > > > > > If you don’t want to or can’t do that (understandable), maybe just let > > > everyone do their own module management on GitHub or whatever, and > > provide > > > a page that links to “add on” modules. (This is the approach > > Elasticsearch > > > takes, I think.) > > > > > > -- Jeremy > > > > > > > > > On Nov 28, 2013, at 11:06, Hari Shreedharan <[email protected] > > > > > wrote: > > > > > > > Juhani and others, > > > > > > > > I agree that it does make sense to add a contrib module to flume > where > > > > non-hadoopy stuff can go. I will start a discussion on this early > next > > > week. > > > > > > > > Hari > > > > > > > > On Thursday, November 28, 2013, Steve Morin wrote: > > > > > > > >> Israel, > > > >> I guess my questions is why the suggestion to use the elastic search > > > >> model, is there something you see that's not working? > > > >> -Steve > > > >> > > > >> > > > >> On Mon, Nov 25, 2013 at 5:34 PM, Israel Ekpo <[email protected] > > > <javascript:;>> > > > >> wrote: > > > >> > > > >>> I think we can take a page of out the ElasticSearch playbook. > > > >>> > > > >>> > > > >>> > > > >> > > > > > > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html > > > >>> > > > >>> I like the model they follow. > > > >>> > > > >>> The Flume architecture makes it easy for plugins at any layer > > (source, > > > >>> interceptor, sink etc) > > > >>> > > > >>> Contributors can host plugins on github and manage the > documentation > > > and > > > >>> maintenance of the plugin. > > > >>> > > > >>> Others can chip it when possible to improve or maintain the > plugins. > > > >>> > > > >>> This will still allow new features to the project without > necessarily > > > >>> meaning that Flume committers are on the hook for maintaining it. > > > >>> > > > >>> > > > >>> > > > >>> *Author and Instructor for the Upcoming Book and Lecture Series* > > > >>> *Massive Log Data Aggregation, Processing, Searching and > > Visualization > > > >> with > > > >>> Open Source Software* > > > >>> *http://massivelogdata.com <http://massivelogdata.com>* > > > >>> > > > >>> > > > >>> On Mon, Nov 25, 2013 at 3:14 AM, Juhani Connolly < > > > >>> [email protected] <javascript:;>> wrote: > > > >>> > > > >>>> Hey guys, > > > >>>> > > > >>>> What I write here is all just my personal opinion and I'm writing > in > > > >>> hopes > > > >>>> of starting a discussion and/or getting feedback. I know I've not > > been > > > >>> very > > > >>>> active on the project recently(due to other engagements) but do > > still > > > >>> want > > > >>>> it to succeed and hope to find more time for it eventually. > > > >>>> > > > >>>> Right now I see new/active issues for the addition of Redis and > > Kafka > > > >>>> sinks, and while they're nice features, I'm personally concerned > > about > > > >>>> feature bloat of the project. There are dozens of interceptors, > > sinks > > > >> and > > > >>>> sources that can be thought of, but most of them are very specific > > to > > > a > > > >>>> specific use-case. > > > >>>> > > > >>>> Every time we add a new component we're also committing to > > maintaining > > > >> it > > > >>>> over future releases, even if the original contributor gets too > busy > > > >> for > > > >>>> it. The more such components get added, the more we will get > > > distracted > > > >>>> from improving core features and getting rid of issues affecting > > them. > > > >>>> > > > >>>> For these reasons I generally haven't submitted components we > > > developed > > > >>>> for internal use(because they are too specific to our use cases), > > just > > > >>>> passing back fixes that fix bugs or apply to the core project. > > > >>>> > > > >>>> For these reasons I think we may want to consider either a) being > > more > > > >>>> selective regarding additional component submissions or b) make a > > > >> contrib > > > >>>> directory to the project which includes the components but doesn't > > > >>>> guarrantee ongoing maintenance or compatibility. > > > >>>> > > > >>>> On the flip side of course, taking approaches like this may > > discourage > > > >>> new > > > >>>> contributors and could thus be considered a negative, and if many > > > >> people > > > >>>> feel this way they should definitely share their thoughts. > > > >>>> > > > >>>> I'd be curious to know what others think, and what direction they > > hope > > > >> to > > > >>>> take the project in the future. > > > >>>> > > > >>> > > > >> > > > > > > > > >
