[
https://issues.apache.org/jira/browse/FLUME-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847251#comment-13847251
]
Ravikumar Visweswara commented on FLUME-1286:
---------------------------------------------
Hello Guys,
Below is my old POC code which does very similar to what you described. There
is an example Topology also.
https://github.com/rvisweswara/flume-storm-connector
Repo also Contains Avro Sink Bolt to push data from Storm to flume.
Limitations Of Spout:
- Multiple instances of spout cant be started on same machine because of port
issues. Can modify the code to add dynamic ports per instance, but its not
ideal.
- In real word cases, Flume Sinks expects a known host and port. But with
storm, flume Avro source can run on any machine (Spout). To use this, one needs
to write their own storm scheduler to run spout on a known IP address.
- Spout can use File channel for persistence. If spout instance is moved to
different machine, messages in the old channel will be left alone.
Code worked fine for poc. But because of above limitations, I ended up writing
a Kafka sink (which I will share shortly) for better reliability
> Storm Sink
> ----------
>
> Key: FLUME-1286
> URL: https://issues.apache.org/jira/browse/FLUME-1286
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Reporter: Mubarak Seyed
> Assignee: Ashish Paliwal
>
> Storm contrib project explains how to push/emit events to storm spout [1]. It
> would be nice to create a flume-storm sink to emit events to storm spout
> output collector [2]
> [1] https://github.com/nathanmarz/storm-contrib/tree/master/storm-scribe
> [2]
> https://github.com/nathanmarz/storm-contrib/blob/master/storm-scribe/src/jvm/storm/scribe/ScribeSpout.java
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
