Not sure if it's useful, but here's a detailed blog post describing how to get the plugins working with an rpm install of Flume 0.9.4:
http://jimter.net/installing-flume-0-9-4-example-plugins/ James -----Original Message----- From: alo alt [mailto:[email protected]] Sent: Thursday, 29 March 2012 12:51 AM To: [email protected] Subject: Re: Integrating Flume a Esper Oh, now we clear. flumeNG (aka old 728 and now 1.1.0-branch) does not support own sinks atm. Here you should take a look into the source code (flume-ng-sinks) to write own plugins, feel free to submit a jira with a review request. Please note, NG isn't ready for production yet. sorry for confusion, we all talk about flume 0.9.4, which will be shipped with cloudera's distribution and support decorators. - Alex -- Alexander Lorenz http://mapredit.blogspot.com On Mar 28, 2012, at 3:43 PM, shekhar sharma wrote: > Hello Alexander, > I have created a class known as EsperSink.java and EsperListener.java, > compiled them and created a jar(esper-sink.jar) file. > > I have put the esper-sink.jar along with all esper related jar into > the lib folder (/flume-1.1.0-incubating-SNAPSHOT/lib) > > And my agent file is something like this: > # Sources > agent1.sources.log.type = exec > agent1.sources.log.command=/usr/bin/tail -F > /home/dev/EsperPOC/SamplePOC/test.txt > agent1.sources.log.channels = log > > # Channgels > agent1.channels.log.type = memory > > # Sinks > agent1.sinks.log.type = Esper > agent1.sinks.log.hostname = namenode > agent1.sinks.log.port = 41414 > agent1.sinks.log.batch-size = 10 > agent1.sinks.log.runner.type = polling > agent1.sinks.log.runner.polling.interval = 5 agent1.sinks.log.channel > = log > > # Load everything > agent1.sources = log > agent1.sinks = log > agent1.channels = log > > > When i am running the agent, it is throwing the following error: > 2012-03-27 20:50:50,317 (conf-file-poller-0) [ERROR - > org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)] > Failed to load configuration data. Exception follows. > org.apache.flume.FlumeException: Unable to load sink type: Esper, > class: Esper > > There is no file with a name flume-conf.xml only property template and this > property template is meant for defining sources and sinks. > Anyways i will create flume-conf.xml and will add the plugins > > Regards, > Som > > On Wed, Mar 28, 2012 at 6:59 PM, alo alt <[email protected]> wrote: > You have to put the them into ./lib (from a flume view) and enable the jars > in flume-conf.xml: > > <property> > <name>flume.plugin.classes</name> > <value>here.the.plugins</value> > <description>Comma separated list of plugin classes</description> > </property> > > -- > Alexander Lorenz > http://mapredit.blogspot.com > > On Mar 28, 2012, at 12:15 PM, shekhar sharma wrote: > > > Hel, > > i have written a custom sink for Esper, can you please how would i make > > FLume to recognize this is my new sink..Do i have to specify in > > configuration file. Can you please help me? > > > > Regards, > > Som > > > > On Wed, Mar 28, 2012 at 1:56 PM, shekhar sharma <[email protected]> > > wrote: > > Thanks for a quick response.. > > But the sinks written for cloudera flume (i was going through hello world > > plugin) use cloudera classes and all.. For FLume-728 the sinks written > > (HDFSSink and IRCSink) is different from cloudera's. > > CLoudera flume requires cloudera's hadoop to be installed, if i am not > > wrong? > > > > My idea is to collect the events from Flume agent and send to Esper, where > > i will query the streams using EPL. > > Can you please tell me what all things are required to write a custom sink. > > i mean what needs to be done so that Flume node recognizes the sink. > > > > Thanks > > > > Regards, > > Som > > > > > > On Wed, Mar 28, 2012 at 1:06 PM, alo alt <[email protected]> wrote: > > Hi, > > > > Cloudera uses in 3u3 Flume-798 > > > > Apache flume isn't different, you can build it among your installation and > > tweak the configs to match them or use the binary distribution: > > https://github.com/cloudera/flume/downloads > > > > As you read, apache flume 1.1.0 was just released: > > http://www.apache.org/dyn/closer.cgi/incubator/flume/ > > > > You have to write plugins for sinks, here you found some created from the > > community: > > https://cwiki.apache.org/FLUME/flume-plugins.html > > > > And the docs: > > http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extend > > ing_via_sink_source_decorator_plugins > > > > Q3 - I don't know, if you use serialized data whats with avro? > > > > - Alex > > > > -- > > Alexander Lorenz > > http://mapredit.blogspot.com > > > > On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote: > > > > > Hello every one, > > > I am trying to integrate FLume and Esper by writing a custom sink for > > > Esper. But as per the documentation , i am unable to do so. i have tried > > > the hello world plugin, but that one also not working. > > > I am using FLume-728. > > > I have few question: > > > > > > Q1. Is Apache Flume is different from Cloudera's Flume? > > > If i am using Cloudera's FLume, then does it have any dependency > > > that i have to use CLoudera's distribution of Hadoop. > > > > > > Q2. What are the necessary steps for writing custom sink ? > > > > > > > > > Q3. I have implemented a POC in Esper, and it takes POJO (Plain old java > > > objects), XML or Map as events. Now if i try integrating both, how > > > feasible it is? > > > > > > Regards, > > > Som Shekhar Sharma > > > > > > > > > > > > > > Information contained in this communication (including any attachments) is confidential and may be privileged or subject to copyright. If you have received this communication in error you are not authorised to use the information in any way and Optiver requests that you notify the sender by return email, destroy all copies and delete the information from your system. Optiver does not represent, warrant or guarantee that this communication is free from computer viruses or other defects or that the integrity of this communication has been maintained. Any views expressed in this communication are those of the individual sender. Optiver does not accept liability for any loss or damage caused directly or indirectly by this communication or its use. Please consider the environment before printing this email.
