Here's a sample config, based on the user guide example //flume.conf =============== # a = agent # c = channel # r = source # k = sink # g = sink group agent.channels = c1 c2 c3 agent.sources = r1 agent.sinks = k1 k2 k3 #agent.sinkgroups =
agent.channels.c1.type = MEMORY agent.channels.c2.type = MEMORY agent.channels.c3.type = MEMORY agent.sources.r1.type = org.example.MySource agent.sources.r1.channels = c1 c2 c3 agent.sources.r1.bind = 0.0.0.0 agent.sources.r1.port = 41414 agent.sources.r1.selector.type = multiplexing agent.sources.r1.selector.header = state agent.sources.r1.selector.mapping.CA = c1 agent.sources.r1.selector.mapping.TX = c2 agent.sources.r1.selector.mapping.HI = c3 agent.sources.r1.selector.default = c1 agent.sinks.k1.type = LOGGER agent.sinks.k1.channel = c1 agent.sinks.k2.type = NULL agent.sinks.k2.channel = c2 agent.sinks.k3.type = org.example.MySink agent.sinks.k3.channel = c3 agent.sinks.k3.mysinkprop = mysinkvalue =============== Cheers, Will On Fri, Apr 20, 2012 at 7:52 AM, Will McQueen <[email protected]> wrote: > Hi Som, > > It sounds like you want to use a multiplexing channel selector and custom > source. Please see the Flume NG User's Guide for discussion of multiplexing > channel selector, under "Fan out flow". > > I'm looking at latest Apache trunk (commit > 37f3597e6fdcbbc7dd43493d0327076868733a09). After I compile the code with > maven, I point my web browser to open: > flume-ng-doc/xhtml/FlumeUserGuide.xhtml > (there's also FlumeDeveloperGuide.xhtml) > > One scenario is that your source component can process each incoming event > and assign a kvp (header and header value) to each one. To assign, you > could use something like: > channelProcessor.processEvent(EventBuilder.withBody(body, headers)); > ...within your source component, and the header you specify would need to > be specified in the config file. Then based on the value of that header, > the event would be directed (based on the header value) to the channel > specified in the config file. > > Another scenario is that the incoming event already contains the header > that your source component needs (eg, when you're using an avro source), in > which case your agent would just propagate the event to the desired > channel, again according to the flume config file. > > Cheers, > Will > > > On Fri, Apr 20, 2012 at 4:30 AM, shekhar sharma <[email protected]>wrote: > >> Hello, >> >> Is it possible to assign multiple channels from a source and send it to >> different sinks in Flume-NG. My use case is something like this: >> (I am currently using trunk) >> >> (A) I have three types of events coming from the web logs lets say >> E1,E2,E3 and ofcourse they can come in any order. Lets say i am using exec >> source which will tail the logs of the log file to the flume source >> >> (B) I will filter the incoming events to the source based on certain >> parameters and send it to designated channel, let's say Ch1, Ch2 and Ch3 >> for E1, E2 and E3. Since i have already done the integration of Esper as >> flume's custom sink, so i think/believe can fire EPL query on the incoming >> events and depending on the type of event E1, E2 or E3, i will send to >> channels (let it be memory channel) Ch1, Ch2, or Ch3. >> >> (3)Then i will have 3 different sinks S1, S2, S3 which will take the >> events from Ch1, Ch2 and Ch3 respectively. >> >> How Can i achieve this using flume-ng? There was one discussion >> http://www.mail-archive.com/[email protected]/msg00617.htmland >> jira >> https://issues.apache.org/jira/browse/FLUME-930. >> >> Regards, >> Som Shekhar Sharma >> >> >> >
