Hello, Is there any way i can configure the source that take the string till the first space as the header?
For example: if i have the log file has ITEM_EVENT LOGIN-29191919fad --- So is there any way i can configure the source that take ITEM_EVENT as header? Regards, Som On Mon, Apr 23, 2012 at 11:44 AM, shekhar sharma <[email protected]>wrote: > Thanks Will. I need to try it. Well my scenario is something like this: > > I have two agents namely agent1 and agent2. > > Agent1 : Source is exec, which tails the output of the log file and Sink > is Avro Sink and channel used is memory. > > Agent 2. Source is Avro source which picks up the events from the Avro > sink from agent1 and send it to Esper Sink.Here also channel is memory. > Basically the Esper sink is used to execute EPL (Event Processing Language) > query on the incoming events, basically filtering the events based on some > parameters. > > Now what i want to achieve is something like this: > Since the logs consist of 3 different types of events, so i will have 3 > different channels so as not to overload the channel. And i will have 3 > different Esper Sinks, which will accumulate the events (a second level > filtering will be employed) > > As you have mentioned that, based on the event headers we can direct the > events to different channels, but in my case headers are empty. So how can > i achieve this, everything is coming as body. Or since i am using exec > source,it takes everything as body? > > Regards, > Som Shekhar > > > > On Fri, Apr 20, 2012 at 9:04 PM, Will McQueen <[email protected]> wrote: > >> Here's a sample config, based on the user guide example >> >> //flume.conf >> =============== >> # a = agent >> # c = channel >> # r = source >> # k = sink >> # g = sink group >> agent.channels = c1 c2 c3 >> agent.sources = r1 >> agent.sinks = k1 k2 k3 >> #agent.sinkgroups = >> >> agent.channels.c1.type = MEMORY >> agent.channels.c2.type = MEMORY >> agent.channels.c3.type = MEMORY >> >> agent.sources.r1.type = org.example.MySource >> agent.sources.r1.channels = c1 c2 c3 >> agent.sources.r1.bind = 0.0.0.0 >> agent.sources.r1.port = 41414 >> agent.sources.r1.selector.type = multiplexing >> agent.sources.r1.selector.header = state >> agent.sources.r1.selector.mapping.CA = c1 >> agent.sources.r1.selector.mapping.TX = c2 >> agent.sources.r1.selector.mapping.HI = c3 >> agent.sources.r1.selector.default = c1 >> >> agent.sinks.k1.type = LOGGER >> agent.sinks.k1.channel = c1 >> >> agent.sinks.k2.type = NULL >> agent.sinks.k2.channel = c2 >> >> agent.sinks.k3.type = org.example.MySink >> agent.sinks.k3.channel = c3 >> agent.sinks.k3.mysinkprop = mysinkvalue >> =============== >> >> >> Cheers, >> Will >> >> >> On Fri, Apr 20, 2012 at 7:52 AM, Will McQueen <[email protected]> wrote: >> >>> Hi Som, >>> >>> It sounds like you want to use a multiplexing channel selector and >>> custom source. Please see the Flume NG User's Guide for discussion of >>> multiplexing channel selector, under "Fan out flow". >>> >>> I'm looking at latest Apache trunk (commit >>> 37f3597e6fdcbbc7dd43493d0327076868733a09). After I compile the code with >>> maven, I point my web browser to open: >>> flume-ng-doc/xhtml/FlumeUserGuide.xhtml >>> (there's also FlumeDeveloperGuide.xhtml) >>> >>> One scenario is that your source component can process each incoming >>> event and assign a kvp (header and header value) to each one. To assign, >>> you could use something like: >>> channelProcessor.processEvent(EventBuilder.withBody(body, headers)); >>> ...within your source component, and the header you specify would need >>> to be specified in the config file. Then based on the value of that header, >>> the event would be directed (based on the header value) to the channel >>> specified in the config file. >>> >>> Another scenario is that the incoming event already contains the header >>> that your source component needs (eg, when you're using an avro source), in >>> which case your agent would just propagate the event to the desired >>> channel, again according to the flume config file. >>> >>> Cheers, >>> Will >>> >>> >>> On Fri, Apr 20, 2012 at 4:30 AM, shekhar sharma >>> <[email protected]>wrote: >>> >>>> Hello, >>>> >>>> Is it possible to assign multiple channels from a source and send it >>>> to different sinks in Flume-NG. My use case is something like this: >>>> (I am currently using trunk) >>>> >>>> (A) I have three types of events coming from the web logs lets say >>>> E1,E2,E3 and ofcourse they can come in any order. Lets say i am using exec >>>> source which will tail the logs of the log file to the flume source >>>> >>>> (B) I will filter the incoming events to the source based on certain >>>> parameters and send it to designated channel, let's say Ch1, Ch2 and Ch3 >>>> for E1, E2 and E3. Since i have already done the integration of Esper as >>>> flume's custom sink, so i think/believe can fire EPL query on the incoming >>>> events and depending on the type of event E1, E2 or E3, i will send to >>>> channels (let it be memory channel) Ch1, Ch2, or Ch3. >>>> >>>> (3)Then i will have 3 different sinks S1, S2, S3 which will take the >>>> events from Ch1, Ch2 and Ch3 respectively. >>>> >>>> How Can i achieve this using flume-ng? There was one discussion >>>> http://www.mail-archive.com/[email protected]/msg00617.htmland >>>> jira >>>> https://issues.apache.org/jira/browse/FLUME-930. >>>> >>>> Regards, >>>> Som Shekhar Sharma >>>> >>>> >>>> >>> >> >
