[
https://issues.apache.org/jira/browse/FLUME-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847969#comment-13847969
]
Gabriel Commeau commented on FLUME-1286:
----------------------------------------
Hi,
We wrote a Flume-to-storm connector, that we’ve been using in production
systems at Comcast for over a year. We presented it as part of our talk about
real-time stream processing at Hadoop World this year. See the slides of the
presentation here (slide 13 most importantly):
http://strataconf.com/stratany2013/public/schedule/detail/30915
I’m still working on open-sourcing it, but I didn’t envision it to be part of
neither Flume nor Storm, as it sits between the two. Indeed, I’m not sure we
want to bring all Storm dependencies into Flume, or vice-versa for Storm. Also,
the way we designed it, there is a Storm sink and a Flume spout, each component
running within its own framework. So I started an independent GitHub project.
I’ll keep you posted as I make progress.
Using a Kafka sink would definitely work, especially because Storm already has
a Kafka spout in storm-contrib. The downside obviously is that you introduce
another set of servers (more maintenance, increased latency, ...).
> Storm Sink
> ----------
>
> Key: FLUME-1286
> URL: https://issues.apache.org/jira/browse/FLUME-1286
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Reporter: Mubarak Seyed
> Assignee: Ashish Paliwal
>
> Storm contrib project explains how to push/emit events to storm spout [1]. It
> would be nice to create a flume-storm sink to emit events to storm spout
> output collector [2]
> [1] https://github.com/nathanmarz/storm-contrib/tree/master/storm-scribe
> [2]
> https://github.com/nathanmarz/storm-contrib/blob/master/storm-scribe/src/jvm/storm/scribe/ScribeSpout.java
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
