If the parameter is chosen from a fixed list, it's cumbersome but can be done. However if you want it to be arbitrary and create topics on demand then you need to write your own code custom sinks and it's not trivial to manage it efficiently
Regards, Gonzalo On Sep 19, 2015 7:42 PM, "Hemanth Abbina" <[email protected]> wrote: > Hi, > > > > I'm new to Flume and thinking to use Flume in the below scenario. > > > > Our system receives events as HTTP POST, and we need to store them in > Kafka(for processing) as well as HDFS(as permanent store). > > > > Can we configure Flume as below ? > > · Source: HTTP (expecting JSON event as HTTP body, with a > dynamic topic name in the URI) > > · Channel: KAFKA (should store the received JSON body, to a topic > mentioned in the URI) > > · Sink: HDFS (should store the data in a folder mentioned in the > URI. > > > > For example, If I receive a JSON event from a HTTP source with the below > attributes, > > · URL: https://xx.xx.xx.xx/event/*abc* > <https://xx.xx.xx.xx/event/abc> > > · Body of POST: { name: xyz, value=123} > > > > The event should be saved to Kafka channel – with topic ‘abc’ and written > to HDFS to a folder as ‘abc’. > > This ‘abc’ will be dynamic and change from event to event. > > > > Is this possible with Flume ? > > > > Thanks in advance > > Hemanth >
