Hi Som,
It sounds like you want to use a multiplexing channel selector and custom
source. Please see the Flume NG User's Guide for discussion of multiplexing
channel selector, under "Fan out flow".
I'm looking at latest Apache trunk (commit
37f3597e6fdcbbc7dd43493d0327076868733a09). After I compile the code with
maven, I point my web browser to open:
flume-ng-doc/xhtml/FlumeUserGuide.xhtml
(there's also FlumeDeveloperGuide.xhtml)
One scenario is that your source component can process each incoming event
and assign a kvp (header and header value) to each one. To assign, you
could use something like:
channelProcessor.processEvent(EventBuilder.withBody(body, headers));
...within your source component, and the header you specify would need to
be specified in the config file. Then based on the value of that header,
the event would be directed (based on the header value) to the channel
specified in the config file.
Another scenario is that the incoming event already contains the header
that your source component needs (eg, when you're using an avro source), in
which case your agent would just propagate the event to the desired
channel, again according to the flume config file.
Cheers,
Will
On Fri, Apr 20, 2012 at 4:30 AM, shekhar sharma <[email protected]>wrote:
> Hello,
>
> Is it possible to assign multiple channels from a source and send it to
> different sinks in Flume-NG. My use case is something like this:
> (I am currently using trunk)
>
> (A) I have three types of events coming from the web logs lets say
> E1,E2,E3 and ofcourse they can come in any order. Lets say i am using exec
> source which will tail the logs of the log file to the flume source
>
> (B) I will filter the incoming events to the source based on certain
> parameters and send it to designated channel, let's say Ch1, Ch2 and Ch3
> for E1, E2 and E3. Since i have already done the integration of Esper as
> flume's custom sink, so i think/believe can fire EPL query on the incoming
> events and depending on the type of event E1, E2 or E3, i will send to
> channels (let it be memory channel) Ch1, Ch2, or Ch3.
>
> (3)Then i will have 3 different sinks S1, S2, S3 which will take the
> events from Ch1, Ch2 and Ch3 respectively.
>
> How Can i achieve this using flume-ng? There was one discussion
> http://www.mail-archive.com/[email protected]/msg00617.htmland
> jira
> https://issues.apache.org/jira/browse/FLUME-930.
>
> Regards,
> Som Shekhar Sharma
>
>
>
