Oh, now we clear. flumeNG (aka old 728 and now 1.1.0-branch) does not support own sinks atm. Here you should take a look into the source code (flume-ng-sinks) to write own plugins, feel free to submit a jira with a review request. Please note, NG isn't ready for production yet.
sorry for confusion, we all talk about flume 0.9.4, which will be shipped with cloudera's distribution and support decorators. - Alex -- Alexander Lorenz http://mapredit.blogspot.com On Mar 28, 2012, at 3:43 PM, shekhar sharma wrote: > Hello Alexander, > I have created a class known as EsperSink.java and EsperListener.java, > compiled them and created a jar(esper-sink.jar) file. > > I have put the esper-sink.jar along with all esper related jar into the lib > folder (/flume-1.1.0-incubating-SNAPSHOT/lib) > > And my agent file is something like this: > # Sources > agent1.sources.log.type = exec > agent1.sources.log.command=/usr/bin/tail -F > /home/dev/EsperPOC/SamplePOC/test.txt > agent1.sources.log.channels = log > > # Channgels > agent1.channels.log.type = memory > > # Sinks > agent1.sinks.log.type = Esper > agent1.sinks.log.hostname = namenode > agent1.sinks.log.port = 41414 > agent1.sinks.log.batch-size = 10 > agent1.sinks.log.runner.type = polling > agent1.sinks.log.runner.polling.interval = 5 > agent1.sinks.log.channel = log > > # Load everything > agent1.sources = log > agent1.sinks = log > agent1.channels = log > > > When i am running the agent, it is throwing the following error: > 2012-03-27 20:50:50,317 (conf-file-poller-0) [ERROR - > org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)] > Failed to load configuration data. Exception follows. > org.apache.flume.FlumeException: Unable to load sink type: Esper, class: Esper > > There is no file with a name flume-conf.xml only property template and this > property template is meant for defining sources and sinks. > Anyways i will create flume-conf.xml and will add the plugins > > Regards, > Som > > On Wed, Mar 28, 2012 at 6:59 PM, alo alt <[email protected]> wrote: > You have to put the them into ./lib (from a flume view) and enable the jars > in flume-conf.xml: > > <property> > <name>flume.plugin.classes</name> > <value>here.the.plugins</value> > <description>Comma separated list of plugin classes</description> > </property> > > -- > Alexander Lorenz > http://mapredit.blogspot.com > > On Mar 28, 2012, at 12:15 PM, shekhar sharma wrote: > > > Hel, > > i have written a custom sink for Esper, can you please how would i make > > FLume to recognize this is my new sink..Do i have to specify in > > configuration file. Can you please help me? > > > > Regards, > > Som > > > > On Wed, Mar 28, 2012 at 1:56 PM, shekhar sharma <[email protected]> > > wrote: > > Thanks for a quick response.. > > But the sinks written for cloudera flume (i was going through hello world > > plugin) use cloudera classes and all.. For FLume-728 the sinks written > > (HDFSSink and IRCSink) is different from cloudera's. > > CLoudera flume requires cloudera's hadoop to be installed, if i am not > > wrong? > > > > My idea is to collect the events from Flume agent and send to Esper, where > > i will query the streams using EPL. > > Can you please tell me what all things are required to write a custom sink. > > i mean what needs to be done so that Flume node recognizes the sink. > > > > Thanks > > > > Regards, > > Som > > > > > > On Wed, Mar 28, 2012 at 1:06 PM, alo alt <[email protected]> wrote: > > Hi, > > > > Cloudera uses in 3u3 Flume-798 > > > > Apache flume isn't different, you can build it among your installation and > > tweak the configs to match them or use the binary distribution: > > https://github.com/cloudera/flume/downloads > > > > As you read, apache flume 1.1.0 was just released: > > http://www.apache.org/dyn/closer.cgi/incubator/flume/ > > > > You have to write plugins for sinks, here you found some created from the > > community: > > https://cwiki.apache.org/FLUME/flume-plugins.html > > > > And the docs: > > http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extending_via_sink_source_decorator_plugins > > > > Q3 - I don't know, if you use serialized data whats with avro? > > > > - Alex > > > > -- > > Alexander Lorenz > > http://mapredit.blogspot.com > > > > On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote: > > > > > Hello every one, > > > I am trying to integrate FLume and Esper by writing a custom sink for > > > Esper. But as per the documentation , i am unable to do so. i have tried > > > the hello world plugin, but that one also not working. > > > I am using FLume-728. > > > I have few question: > > > > > > Q1. Is Apache Flume is different from Cloudera's Flume? > > > If i am using Cloudera's FLume, then does it have any dependency > > > that i have to use CLoudera's distribution of Hadoop. > > > > > > Q2. What are the necessary steps for writing custom sink ? > > > > > > > > > Q3. I have implemented a POC in Esper, and it takes POJO (Plain old java > > > objects), XML or Map as events. Now if i try integrating both, how > > > feasible it is? > > > > > > Regards, > > > Som Shekhar Sharma > > > > > > > > > > > > > >
