> If you plan to use NiFi for the long term, it might be worth investing in > converting your custom Flume components to NiFi processors. We can help you > get started if you need any guidance going that route.
+1. Running Flume sources/sinks is meant as a transition step. It's really useful if you have a complex Flume flow and want to migrate only parts of it over to NiFi at a time. I would port any custom sources and sinks to NiFi once you knew that it would meet your needs well. NiFi has a lot of documentation on writing processors and the concepts map pretty well if you're already familiar with Flume's execution model. -Joey On Thu, Oct 8, 2015 at 9:48 AM, Bryan Bende <bbe...@gmail.com> wrote: > > Hi Parul, > > It is possible to deploy a custom Flume source/sink to NiFi, but due to the > way the Flume processors load the classes for the sources and sinks, the jar > you deploy to the lib directory also needs to include the other dependencies > your source/sink needs (or they each need to individually be in lib/ > directly). > > So here is a sample project I created that makes a shaded jar: > https://github.com/bbende/my-flume-source > > It will contain the custom source and following dependencies all in one jar: > > org.apache.flume:my-flume-source:jar:1.0-SNAPSHOT > +- org.apache.flume:flume-ng-sdk:jar:1.6.0:compile > +- org.apache.flume:flume-ng-core:jar:1.6.0:compile > +- org.apache.flume:flume-ng-configuration:jar:1.6.0:compile > +- org.apache.flume:flume-ng-auth:jar:1.6.0:compile > \- com.google.guava:guava:jar:11.0.2:compile > \- com.google.code.findbugs:jsr305:jar:1.3.9:compile > > I copied that to NiFi lib, restarted, created an ExecuteFlumeSource processor > with the following config: > > Source Type = org.apache.flume.MySource > Agent Name = a1 > Source Name = r1 > Flume Configuration = a1.sources = r1 > > And I was getting the output in nifi/logs/nifi-bootstrap.log > > Keep in mind that this could become risky because any classes found in the > lib directory would be accessible to all NARs in NiFi and would be found > before classes within a NAR because the parent is checked first during class > loading. This example isn't too risky because we are only bringing in flume > jars and one guava jar, but for example if another nar uses a different > version of guava this is going to cause a problem. > > If you plan to use NiFi for the long term, it might be worth investing in > converting your custom Flume components to NiFi processors. We can help you > get started if you need any guidance going that route. > > -Bryan > > > On Thu, Oct 8, 2015 at 2:30 AM, Parul Agrawal <parulagrawa...@gmail.com> > wrote: >> >> Hello Bryan, >> >> Thank you very much for your response. >> >> Is it possible to have customized flume source and sink in Nifi? >> I have my own customized source and sink? I followed below steps to add my >> own customized source but it did not work. >> >> 1) Created Maven project and added customized source. (flume.jar was created >> after this step) >> 2) Added flume.jar file to nifi-0.3.0/lib folder. >> 3) Added flume source processor with the below configuration >> >> Property Value >> Source Type com.flume.source.Source >> Agent Name a1 >> Source Name k1. >> >> But I am getting the below error in Flume Source Processor. >> "Failed to run validation due to java.lang.NoClassDefFoundError : >> /org/apache/flume/PollableSource." >> >> Can you please help me in this regard. Any step/configuration I missed. >> >> Thanks and Regards, >> Parul >> >> >> On Wed, Oct 7, 2015 at 6:57 PM, Bryan Bende <bbe...@gmail.com> wrote: >>> >>> Hello, >>> >>> The NiFi Flume processors are for running Flume sources and sinks with in >>> NiFi. They don't communicate with an external Flume process. >>> >>> In your example you would need an ExecuteFlumeSource configured to run the >>> netcat source, connected to a ExecuteFlumeSink configured with the logger. >>> >>> -Bryan >>> >>> On Wednesday, October 7, 2015, Parul Agrawal <parulagrawa...@gmail.com> >>> wrote: >>>> >>>> Hi, >>>> >>>> I was trying to run Nifi Flume processor with the below mentioned >>>> details but not could bring it up. >>>> >>>> I already started flume with the sample configuration file >>>> ============================================= >>>> # example.conf: A single-node Flume configuration >>>> >>>> # Name the components on this agent >>>> a1.sources = r1 >>>> a1.sinks = k1 >>>> a1.channels = c1 >>>> >>>> # Describe/configure the source >>>> a1.sources.r1.type = netcat >>>> a1.sources.r1.bind = localhost >>>> a1.sources.r1.port = 44444 >>>> >>>> # Describe the sink >>>> a1.sinks.k1.type = logger >>>> >>>> # Use a channel which buffers events in memory >>>> a1.channels.c1.type = memory >>>> a1.channels.c1.capacity = 1000 >>>> a1.channels.c1.transactionCapacity = 100 >>>> >>>> # Bind the source and sink to the channel >>>> a1.sources.r1.channels = c1 >>>> a1.sinks.k1.channel = c1 >>>> ============================================= >>>> >>>> Command used to start flume : $ bin/flume-ng agent --conf conf >>>> --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console >>>> >>>> In the Nifi browser of ExecuteFlumeSink following configuration was done: >>>> Property Value >>>> Sink Type logger >>>> Agent Name a1 >>>> Sink Name k1. >>>> >>>> Event is sent to the flume using: >>>> $ telnet localhost 44444 >>>> Trying 127.0.0.1... >>>> Connected to localhost.localdomain (127.0.0.1). >>>> Escape character is '^]'. >>>> Hello world! <ENTER> >>>> OK >>>> >>>> But I could not get any data in the nifi flume processor. Request your >>>> help in this. >>>> Do i need to change the example.conf file of flume so that Nifi Flume >>>> Sink should get the data. >>>> >>>> Thanks and Regards, >>>> Parul >>> >>> >>> >>> -- >>> Sent from Gmail Mobile >> >> >
