Hello,
Can you tell us what you are trying to route on in the json? What regular
expression did you try in RouteOnContent?
-Bryan
On Monday, October 26, 2015, Parul Agrawal <parulagrawa...@gmail.com> wrote:
> Hi,
>
> Thank you very much for all the support.
> I have written a custom processor to split json to multiple json.
> Now I would like to route the flowfile based on the content of the
> flowfile.
> I tried using RouteOnContent. But it did not work.
>
> Can you please help me how can i route the flowfile based on the
> content/data it contains.
>
> Thanks and Regards,
> Parul
>
>
>
> On Tue, Oct 13, 2015 at 6:54 PM, Bryan Bende <bbe...@gmail.com
> <javascript:_e(%7B%7D,'cvml','bbe...@gmail.com');>> wrote:
>
>> Parul,
>>
>> You can use SplitJson to take a large JSON document and split an array
>> element into individual documents. I took the json you attached and created
>> a flow like GetFile -> SplitJson -> SplitJson -> PutFile
>>
>> In the first SplitJson the path I used was $.packet.proto and in the
>> second I used $.field This seemed to mostly work except some of the splits
>> going into PutFile still have another level of "field" which needs to be
>> split again so would possibly need some conditional logic to split certain
>> documents again.
>>
>> Alternatively you could write a custom processor that restructures your
>> JSON.
>>
>> -Bryan
>>
>>
>>
>> On Tue, Oct 13, 2015 at 8:36 AM, Parul Agrawal <parulagrawa...@gmail.com
>> <javascript:_e(%7B%7D,'cvml','parulagrawa...@gmail.com');>> wrote:
>>
>>> Hi,
>>>
>>> I tried with the above json element. But I am getting the below
>>> mentioned error:
>>>
>>> 2015-10-12 23:53:39,209 ERROR [Timer-Driven Process Thread-9]
>>> o.a.n.p.standard.ConvertJSONToSQL
>>> ConvertJSONToSQL[id=0e964781-6914-486f-8bb7-214c6a1cd66e] Failed to parse
>>> StandardFlowFileRecord[uuid=dfc16db0-c7a6-4e9e-8b4d-8c5b4ec50742,claim=StandardContentClaim
>>> [resourceClaim=StandardResourceClaim[id=1444483036971-1, container=default,
>>> section=1], offset=132621, length=55],offset=0,name=json,size=55] as JSON
>>> due to org.apache.nifi.processor.exception.ProcessException: IOException
>>> thrown from ConvertJSONToSQL[id=0e964781-6914-486f-8bb7-214c6a1cd66e]:
>>> org.codehaus.jackson.JsonParseException: Unexpected character ('I' (code
>>> 73)): expected a valid value (number, String, array, object, 'true',
>>> 'false' or 'null')
>>>
>>> Also I have a huge json object attached (new.json). Can you guide me on
>>> how do i use ConvertJSONToSQL processor.
>>> Should I use any other processor before using ConvertJSONToSQL processor
>>> so that this new.json can be converted in to a flat document of
>>> key/value pairs, or an array of flat documents.
>>>
>>> Any help/guidance would be really useful.
>>>
>>> Thanks and Regards,
>>> Parul
>>>
>>> On Mon, Oct 12, 2015 at 10:36 PM, Bryan Bende <bbe...@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','bbe...@gmail.com');>> wrote:
>>>
>>>> I think ConvertJSONToSQL expects a flat document of key/value pairs, or
>>>> an array of flat documents. So I think your JSON would be:
>>>>
>>>> [
>>>> {"firstname":"John", "lastname":"Doe"},
>>>> {"firstname":"Anna", "lastname":"Smith"}
>>>> ]
>>>>
>>>> The table name will come from the Table Name property.
>>>>
>>>> Let us know if this doesn't work.
>>>>
>>>> -Bryan
>>>>
>>>>
>>>> On Mon, Oct 12, 2015 at 12:19 PM, Parul Agrawal <
>>>> parulagrawa...@gmail.com
>>>> <javascript:_e(%7B%7D,'cvml','parulagrawa...@gmail.com');>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Thank you very much for all the support.
>>>>> I could able to convert XML format to json using custom flume source.
>>>>>
>>>>> Now I would need ConvertJSONToSQL processor to insert data into SQL.
>>>>> I am trying to get hands-on on this processor. Will update you on this.
>>>>> Meanwhile if any example you could share to use this processor for a
>>>>> sample
>>>>> json data, then it would be great.
>>>>>
>>>>> ===============
>>>>>
>>>>> 1) I tried using ConvertJSONToSQL processor with the below sample json
>>>>> file:
>>>>>
>>>>> "details":[
>>>>> {"firstname":"John", "lastname":"Doe"},
>>>>> {"firstname":"Anna", "lastname":"Smith"}
>>>>> ]
>>>>>
>>>>> 2) I created table *details *in the postgreSQL
>>>>> * select * from details ;*
>>>>> * firstname | lastname*
>>>>> *-----------+----------*
>>>>> *(0 rows)*
>>>>>
>>>>> 3) ConvertJSONToSQL Processor property details are as below:
>>>>> *Property * *Value*
>>>>> JDBC Connection PoolInfo DBCPConnectionPool
>>>>> Statement TypeInfo INSERT
>>>>> Table NameInfo details
>>>>> Catalog NameInfo No value set
>>>>> Translate Field NamesInfo false
>>>>> Unmatched Field BehaviorInfo Ignore Unmatched Fields
>>>>> Update KeysInfo No value set
>>>>>
>>>>> But I am getting the below mentioned error in ConvertJSONToSQL
>>>>> Processor.
>>>>> 2015-10-12 05:15:19,584 ERROR [Timer-Driven Process Thread-1]
>>>>> o.a.n.p.standard.ConvertJSONToSQL
>>>>> ConvertJSONToSQL[id=0e964781-6914-486f-8bb7-214c6a1cd66e] Failed to
>>>>> convert
>>>>> StandardFlowFileRecord[uuid=3a58716b-1474-4d75-91c1-e2fc3b9175ba,claim=StandardContentClaim
>>>>> [resourceClaim=StandardResourceClaim[id=1444483036971-1,
>>>>> container=default,
>>>>> section=1], offset=115045, length=104],offset=0,name=json,size=104] to a
>>>>> SQL INSERT statement due to
>>>>> org.apache.nifi.processor.exception.ProcessException: None of the fields
>>>>> in
>>>>> the JSON map to the columns defined by the details table; routing to
>>>>> failure: org.apache.nifi.processor.exception.ProcessException: None of the
>>>>> fields in the JSON map to the columns defined by the details table
>>>>>
>>>>> Thanks and Regards,
>>>>> Parul
>>>>>
>>>>> On Sat, Oct 10, 2015 at 9:45 PM, Joey Echeverria <joe...@gmail.com
>>>>> <javascript:_e(%7B%7D,'cvml','joe...@gmail.com');>> wrote:
>>>>>
>>>>>> I've done something like this by wrapping the command in a shell
>>>>>> script:
>>>>>>
>>>>>> http://ingest.tips/2014/12/22/getting-started-with-apache-nifi/
>>>>>>
>>>>>> My use case was slightly different, but I'm pretty sure you can adapt
>>>>>> the same idea.
>>>>>>
>>>>>> -Joey
>>>>>>
>>>>>> On Oct 10, 2015, at 03:52, Parul Agrawal <parulagrawa...@gmail.com
>>>>>> <javascript:_e(%7B%7D,'cvml','parulagrawa...@gmail.com');>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I actually need to get the data from pipe.
>>>>>> So the actual command I would need is mkfifo /tmp/packet;tshark -i
>>>>>> ens160 -T pdml >/tmp/packet.
>>>>>> Is it possible to use ExecuteProcessor for multiple commands ?
>>>>>>
>>>>>> On Sat, Oct 10, 2015 at 1:04 PM, Parul Agrawal <
>>>>>> parulagrawa...@gmail.com
>>>>>> <javascript:_e(%7B%7D,'cvml','parulagrawa...@gmail.com');>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I added custom flume source and when flume source is sending the
>>>>>>> data to flume sink, below mentioned error is thrown at flume sink.
>>>>>>>
>>>>>>> Administratively Yielded for 1 sec due to processing failure
>>>>>>> 2015-10-10 02:30:45,027 WARN [Timer-Driven Process Thread-9]
>>>>>>> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
>>>>>>> ExecuteFlumeSink[id=2d08dfe7-4fd1-4a10-9d25-0b007a2c41bf] due to
>>>>>>> uncaught
>>>>>>> Exception: java.lang.IllegalStateException: close() called when
>>>>>>> transaction
>>>>>>> is OPEN - you must either commit or rollback first
>>>>>>> 2015-10-10 02:30:45,028 WARN [Timer-Driven Process Thread-9]
>>>>>>> o.a.n.c.t.ContinuallyRunProcessorTask
>>>>>>> java.lang.IllegalStateException: close() called when transaction is
>>>>>>> OPEN - you must either commit or rollback first
>>>>>>> at
>>>>>>> com.google.common.base.Preconditions.checkState(Preconditions.java:172)
>>>>>>> ~[guava-r05.jar:na]
>>>>>>> at
>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
>>>>>>> ~[flume-ng-core-1.6.0.jar:1.6.0]
>>>>>>> at
>>>>>>> org.apache.flume.sink.LoggerSink.process(LoggerSink.java:105)
>>>>>>> ~[flume-ng-core-1.6.0.jar:1.6.0]
>>>>>>> at
>>>>>>> org.apache.nifi.processors.flume.ExecuteFlumeSink.onTrigger(ExecuteFlumeSink.java:139)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.apache.nifi.processors.flume.AbstractFlumeProcessor.onTrigger(AbstractFlumeProcessor.java:148)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077)
>>>>>>> ~[nifi-framework-core-0.3.0.jar:0.3.0]
>>>>>>> at
>>>>>>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127)
>>>>>>> [nifi-framework-core-0.3.0.jar:0.3.0]
>>>>>>> at
>>>>>>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49)
>>>>>>> [nifi-framework-core-0.3.0.jar:0.3.0]
>>>>>>> at
>>>>>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
>>>>>>> [nifi-framework-core-0.3.0.jar:0.3.0]
>>>>>>> at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> [na:1.7.0_85]
>>>>>>> at
>>>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>> [na:1.7.0_85]
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>> [na:1.7.0_85]
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>> [na:1.7.0_85]
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> [na:1.7.0_85]
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> [na:1.7.0_85]
>>>>>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]
>>>>>>> 2015-10-10 02:30:46,029 ERROR [Timer-Driven Process Thread-9]
>>>>>>> o.a.n.processors.flume.ExecuteFlumeSink
>>>>>>> ExecuteFlumeSink[id=2d08dfe7-4fd1-4a10-9d25-0b007a2c41bf]
>>>>>>> ExecuteFlumeSink[id=2d08dfe7-4fd1-4a10-9d25-0b007a2c41bf] failed to
>>>>>>> process
>>>>>>> due to org.apache.nifi.processor.exception.FlowFileHandlingException:
>>>>>>> StandardFlowFileRecord[uuid=8832b036-51a4-49cf-9703-fc4ed443ab80,claim=StandardContentClaim
>>>>>>> [resourceClaim=StandardResourceClaim[id=1444462207782-7,
>>>>>>> container=default,
>>>>>>> section=7], offset=180436,
>>>>>>> length=14078],offset=0,name=8311685679474355,size=14078] is not known in
>>>>>>> this session (StandardProcessSession[id=218318]); rolling back session:
>>>>>>> org.apache.nifi.processor.exception.FlowFileHandlingException:
>>>>>>> StandardFlowFileRecord[uuid=8832b036-51a4-49cf-9703-fc4ed443ab80,claim=StandardContentClaim
>>>>>>> [resourceClaim=StandardResourceClaim[id=1444462207782-7,
>>>>>>> container=default,
>>>>>>> section=7], offset=180436,
>>>>>>> length=14078],offset=0,name=8311685679474355,size=14078] is not known in
>>>>>>> this session (StandardProcessSession[id=218318])
>>>>>>>
>>>>>>> Any idea what could be wrong in this.
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Parul
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 9, 2015 at 6:32 PM, Bryan Bende <bbe...@gmail.com
>>>>>>> <javascript:_e(%7B%7D,'cvml','bbe...@gmail.com');>> wrote:
>>>>>>>
>>>>>>>> Hi Parul,
>>>>>>>>
>>>>>>>> I think it would be good to keep the convo going on the users list
>>>>>>>> since there are more people who can offer help there, and also helps
>>>>>>>> everyone learn new solutions.
>>>>>>>>
>>>>>>>> The quick answer though is that NiFi has an ExecuteProcess
>>>>>>>> processor which could execute "tshark -i eth0 -T pdml".
>>>>>>>>
>>>>>>>> There is not currently an XmlToJson processor, so this could be a
>>>>>>>> place where you need a custom processor. For simple cases you can use
>>>>>>>> an
>>>>>>>> EvaluateXPath processor to extract values from the XML, and then a
>>>>>>>> ReplaceText processor to build a new json document from those extracted
>>>>>>>> values.
>>>>>>>>
>>>>>>>> -Bryan
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 9, 2015 at 3:39 AM, Parul Agrawal <
>>>>>>>> parulagrawa...@gmail.com
>>>>>>>> <javascript:_e(%7B%7D,'cvml','parulagrawa...@gmail.com');>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Little more to add.....
>>>>>>>>> I need to keep reading the flowfile till END_TAG is received.
>>>>>>>>> i.e. we may need to concatenate the flowfile data till END_TAG.
>>>>>>>>> and then convert it to json and call PutFile() processor.
>>>>>>>>>
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Parul
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Oct 9, 2015 at 10:56 AM, Parul Agrawal <
>>>>>>>>> parulagrawa...@gmail.com
>>>>>>>>> <javascript:_e(%7B%7D,'cvml','parulagrawa...@gmail.com');>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Thank you very much again for the guidance provided.
>>>>>>>>>> Basically I would need a processor which would convert XML file
>>>>>>>>>> to Json.
>>>>>>>>>>
>>>>>>>>>> Currently I have a flume source which is of type "exec" and the
>>>>>>>>>> command used is "tshark -i eth0 -T pdml".
>>>>>>>>>>
>>>>>>>>>> Here Flume source keeps sending data to flume sink. This flow
>>>>>>>>>> file would be of PDML format.
>>>>>>>>>>
>>>>>>>>>> Now I need a processor which would do the following
>>>>>>>>>>
>>>>>>>>>> 1) Form a complete XML file based on START TAG (<packet>)
>>>>>>>>>> and END TAG (</packet>)
>>>>>>>>>> 2) Once the XML message is formed convert it to json.
>>>>>>>>>> 3) Place a json file to local directory using PutFile() processor.
>>>>>>>>>>
>>>>>>>>>> I am not sure if I could able to explain the processor
>>>>>>>>>> requirement.
>>>>>>>>>> Would be really great if you could help me in this.
>>>>>>>>>>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>> Parul
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Oct 8, 2015 at 10:02 PM, Joey Echeverria <
>>>>>>>>>> joe...@gmail.com
>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','joe...@gmail.com');>> wrote:
>>>>>>>>>>
>>>>>>>>>>> > If you plan to use NiFi for the long term, it might be worth
>>>>>>>>>>> investing in converting your custom Flume components to NiFi
>>>>>>>>>>> processors. We
>>>>>>>>>>> can help you get started if you need any guidance going that route.
>>>>>>>>>>>
>>>>>>>>>>> +1. Running Flume sources/sinks is meant as a transition step.
>>>>>>>>>>> It's
>>>>>>>>>>> really useful if you have a complex Flume flow and want to
>>>>>>>>>>> migrate
>>>>>>>>>>> only parts of it over to NiFi at a time. I would port any custom
>>>>>>>>>>> sources and sinks to NiFi once you knew that it would meet your
>>>>>>>>>>> needs
>>>>>>>>>>> well. NiFi has a lot of documentation on writing processors and
>>>>>>>>>>> the
>>>>>>>>>>> concepts map pretty well if you're already familiar with Flume's
>>>>>>>>>>> execution model.
>>>>>>>>>>>
>>>>>>>>>>> -Joey
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Oct 8, 2015 at 9:48 AM, Bryan Bende <bbe...@gmail.com
>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','bbe...@gmail.com');>> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> > Hi Parul,
>>>>>>>>>>> >
>>>>>>>>>>> > It is possible to deploy a custom Flume source/sink to NiFi,
>>>>>>>>>>> but due to the way the Flume processors load the classes for the
>>>>>>>>>>> sources
>>>>>>>>>>> and sinks, the jar you deploy to the lib directory also needs to
>>>>>>>>>>> include
>>>>>>>>>>> the other dependencies your source/sink needs (or they each need to
>>>>>>>>>>> individually be in lib/ directly).
>>>>>>>>>>> >
>>>>>>>>>>> > So here is a sample project I created that makes a shaded jar:
>>>>>>>>>>> > https://github.com/bbende/my-flume-source
>>>>>>>>>>> >
>>>>>>>>>>> > It will contain the custom source and following dependencies
>>>>>>>>>>> all in one jar:
>>>>>>>>>>> >
>>>>>>>>>>> > org.apache.flume:my-flume-source:jar:1.0-SNAPSHOT
>>>>>>>>>>> > +- org.apache.flume:flume-ng-sdk:jar:1.6.0:compile
>>>>>>>>>>> > +- org.apache.flume:flume-ng-core:jar:1.6.0:compile
>>>>>>>>>>> > +- org.apache.flume:flume-ng-configuration:jar:1.6.0:compile
>>>>>>>>>>> > +- org.apache.flume:flume-ng-auth:jar:1.6.0:compile
>>>>>>>>>>> > \- com.google.guava:guava:jar:11.0.2:compile
>>>>>>>>>>> > \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
>>>>>>>>>>> >
>>>>>>>>>>> > I copied that to NiFi lib, restarted, created an
>>>>>>>>>>> ExecuteFlumeSource processor with the following config:
>>>>>>>>>>> >
>>>>>>>>>>> > Source Type = org.apache.flume.MySource
>>>>>>>>>>> > Agent Name = a1
>>>>>>>>>>> > Source Name = r1
>>>>>>>>>>> > Flume Configuration = a1.sources = r1
>>>>>>>>>>> >
>>>>>>>>>>> > And I was getting the output in nifi/logs/nifi-bootstrap.log
>>>>>>>>>>> >
>>>>>>>>>>> > Keep in mind that this could become risky because any classes
>>>>>>>>>>> found in the lib directory would be accessible to all NARs in NiFi
>>>>>>>>>>> and
>>>>>>>>>>> would be found before classes within a NAR because the parent is
>>>>>>>>>>> checked
>>>>>>>>>>> first during class loading. This example isn't too risky because we
>>>>>>>>>>> are
>>>>>>>>>>> only bringing in flume jars and one guava jar, but for example if
>>>>>>>>>>> another
>>>>>>>>>>> nar uses a different version of guava this is going to cause a
>>>>>>>>>>> problem.
>>>>>>>>>>> >
>>>>>>>>>>> > If you plan to use NiFi for the long term, it might be worth
>>>>>>>>>>> investing in converting your custom Flume components to NiFi
>>>>>>>>>>> processors. We
>>>>>>>>>>> can help you get started if you need any guidance going that route.
>>>>>>>>>>> >
>>>>>>>>>>> > -Bryan
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > On Thu, Oct 8, 2015 at 2:30 AM, Parul Agrawal <
>>>>>>>>>>> parulagrawa...@gmail.com
>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','parulagrawa...@gmail.com');>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> >>
>>>>>>>>>>> >> Hello Bryan,
>>>>>>>>>>> >>
>>>>>>>>>>> >> Thank you very much for your response.
>>>>>>>>>>> >>
>>>>>>>>>>> >> Is it possible to have customized flume source and sink in
>>>>>>>>>>> Nifi?
>>>>>>>>>>> >> I have my own customized source and sink? I followed below
>>>>>>>>>>> steps to add my own customized source but it did not work.
>>>>>>>>>>> >>
>>>>>>>>>>> >> 1) Created Maven project and added customized source.
>>>>>>>>>>> (flume.jar was created after this step)
>>>>>>>>>>> >> 2) Added flume.jar file to nifi-0.3.0/lib folder.
>>>>>>>>>>> >> 3) Added flume source processor with the below configuration
>>>>>>>>>>> >>
>>>>>>>>>>> >> Property Value
>>>>>>>>>>> >> Source Type com.flume.source.Source
>>>>>>>>>>> >> Agent Name a1
>>>>>>>>>>> >> Source Name k1.
>>>>>>>>>>> >>
>>>>>>>>>>> >> But I am getting the below error in Flume Source Processor.
>>>>>>>>>>> >> "Failed to run validation due to
>>>>>>>>>>> java.lang.NoClassDefFoundError : /org/apache/flume/PollableSource."
>>>>>>>>>>> >>
>>>>>>>>>>> >> Can you please help me in this regard. Any step/configuration
>>>>>>>>>>> I missed.
>>>>>>>>>>> >>
>>>>>>>>>>> >> Thanks and Regards,
>>>>>>>>>>> >> Parul
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >> On Wed, Oct 7, 2015 at 6:57 PM, Bryan Bende <bbe...@gmail.com
>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','bbe...@gmail.com');>> wrote:
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> Hello,
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> The NiFi Flume processors are for running Flume sources and
>>>>>>>>>>> sinks with in NiFi. They don't communicate with an external Flume
>>>>>>>>>>> process.
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> In your example you would need an ExecuteFlumeSource
>>>>>>>>>>> configured to run the netcat source, connected to a ExecuteFlumeSink
>>>>>>>>>>> configured with the logger.
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> -Bryan
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> On Wednesday, October 7, 2015, Parul Agrawal <
>>>>>>>>>>> parulagrawa...@gmail.com
>>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','parulagrawa...@gmail.com');>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> Hi,
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> I was trying to run Nifi Flume processor with the below
>>>>>>>>>>> mentioned
>>>>>>>>>>> >>>> details but not could bring it up.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> I already started flume with the sample configuration file
>>>>>>>>>>> >>>> =============================================
>>>>>>>>>>> >>>> # example.conf: A single-node Flume configuration
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> # Name the components on this agent
>>>>>>>>>>> >>>> a1.sources = r1
>>>>>>>>>>> >>>> a1.sinks = k1
>>>>>>>>>>> >>>> a1.channels = c1
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> # Describe/configure the source
>>>>>>>>>>> >>>> a1.sources.r1.type = netcat
>>>>>>>>>>> >>>> a1.sources.r1.bind = localhost
>>>>>>>>>>> >>>> a1.sources.r1.port = 44444
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> # Describe the sink
>>>>>>>>>>> >>>> a1.sinks.k1.type = logger
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> # Use a channel which buffers events in memory
>>>>>>>>>>> >>>> a1.channels.c1.type = memory
>>>>>>>>>>> >>>> a1.channels.c1.capacity = 1000
>>>>>>>>>>> >>>> a1.channels.c1.transactionCapacity = 100
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> # Bind the source and sink to the channel
>>>>>>>>>>> >>>> a1.sources.r1.channels = c1
>>>>>>>>>>> >>>> a1.sinks.k1.channel = c1
>>>>>>>>>>> >>>> =============================================
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> Command used to start flume : $ bin/flume-ng agent --conf
>>>>>>>>>>> conf
>>>>>>>>>>> >>>> --conf-file example.conf --name a1
>>>>>>>>>>> -Dflume.root.logger=INFO,console
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> In the Nifi browser of ExecuteFlumeSink following
>>>>>>>>>>> configuration was done:
>>>>>>>>>>> >>>> Property Value
>>>>>>>>>>> >>>> Sink Type logger
>>>>>>>>>>> >>>> Agent Name a1
>>>>>>>>>>> >>>> Sink Name k1.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> Event is sent to the flume using:
>>>>>>>>>>> >>>> $ telnet localhost 44444
>>>>>>>>>>> >>>> Trying 127.0.0.1...
>>>>>>>>>>> >>>> Connected to localhost.localdomain (127.0.0.1).
>>>>>>>>>>> >>>> Escape character is '^]'.
>>>>>>>>>>> >>>> Hello world! <ENTER>
>>>>>>>>>>> >>>> OK
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> But I could not get any data in the nifi flume processor.
>>>>>>>>>>> Request your
>>>>>>>>>>> >>>> help in this.
>>>>>>>>>>> >>>> Do i need to change the example.conf file of flume so that
>>>>>>>>>>> Nifi Flume
>>>>>>>>>>> >>>> Sink should get the data.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> Thanks and Regards,
>>>>>>>>>>> >>>> Parul
>>>>>>>>>>> >>>
>>>>>>>>>>> >>>
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> --
>>>>>>>>>>> >>> Sent from Gmail Mobile
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
Sent from Gmail Mobile
