Re: PutParquet with S3

s-2.7.3.jar > hadoop-common-2.7.3.jar > httpclient-4.5.3.jar > httpcore-4.4.4.jar > jackson-annotations-2.6.0.jar > jackson-core-2.6.1.jar > jackson-databind-2.6.1.jar > > So just grab those from maven central and you should be good to go. > > -joey > > On Dec 5

PutParquet with S3

Hi Is it possible to use PutParquet processor to write files into S3? I tried by setting s3 bucket in core-site.xml file but i am getting *No FileSystem for scheme: s3a* *core-site.xml* fs.defaultFS s3a://testing fs.s3a.access.key fs.s3a.secret.key

Netflow parser

Is there any Processor available for Netflow? If not what is the best way to get Netflow data parsed using nifi?

Re: Syslog timestamp

Any help? On Thu, Aug 4, 2016 at 5:25 AM, Madhukar Thota <madhukar.th...@gmail.com> wrote: > How to convert Both RFC5424 and RFC3164 syslog timestamp attribute to > type to long. > > Will some thing like this works? > > ${syslog.timestamp:toDate('M

Syslog timestamp

How to convert Both RFC5424 and RFC3164 syslog timestamp attribute to type to long. Will some thing like this works? ${syslog.timestamp:toDate('MMM d HH:mm:ss'):toNumber():or( ${syslog.timestamp:toDate("-MM-dd'T'HH:mm:ss.SZ"):toNumber()} ):or(

Re: Syslog to avro format

one (that included ints). > > HTH, > > Conrad > > > > *From: *Madhukar Thota <madhukar.th...@gmail.com> > *Reply-To: *"users@nifi.apache.org" <users@nifi.apache.org> > *Date: *Tuesday, 26 July 2016 at 09:52 > *To: *"users@nifi.apache.

Syslog to avro format

Friends, What is the best way to get Syslog data into avro format without converting to JSON? Any suggestions?

Re: Json Split

t to extract $.json ? >> >> In your original example only the second document had a field called >> "json", and the matched relationship coming out of EvaluateJsonPath will >> only receive the json documents that had the path being extracted. >> >> -Br

Re: JSON Schema

; log.error("Doesn't adhere to schema", ve) > valid = false > } > } as InputStreamCallback) > > session.transfer(flowFile, valid ? REL_SUCCESS : REL_FAILURE) > > > Hope this helps! > > Regards, > Matt > > [1] https://issues.ap

Re: Json Split

cuments. But i need to > only one document > > "{"json":"data","extracted":"from","message":"payload"}" > > How can i get that? > > On Tue, May 17, 2016 at 12:21 PM, Bryan Bende <bbe...@gmail.com> wrote: > >>

Re: Json Split

de <bbe...@gmail.com> wrote: > Hello, > > I think this would probably be better handled by SplitText with a line > count of 1. > > SplitJson would be more for splitting an array of JSON documents, or a > field that is an array. > > -Bryan > > On Tue, May 17,

Json Split

I have a incoming json from kafka with two documents seperated by new line {"index":{"_index":"mylogger-2014.06.05","_type":"mytype-host.domain.com"}}{"json":"data","extracted":"from","message":"payload"} I want to get the second document after new line. How can i split the json by new line

JSON Schema

is it possible to do validation of incoming json via http Processor with Json Schema in nifi? Example Json: { name: "Test", timestamp: 1463499695, tags: { "host": "Test_1", "ip" : "1.1.1.1" }, fields: { "cpu": 10.2, "load: : 15.6 } } JSON schema: "type":

FileSize

Friends, Is it possible to set file size like 500 mb or 1 GB before writing to hdfs? I want to write large files instead of lot of smaller files to hdfs? If possible, what processor i need to use to achive the size? -Madhu

Re: Lua usage in ExecuteScript Processor

bottom line is that LuaJ should probably only be used to leverage > business logic written in Lua, not Java. If you want access to Java > libraries, I'd use another script engine such as Groovy. > > Regards, > Matt > > On Wed, May 4, 2016 at 11:39 AM, Madhukar Thota > &l

Re: Lua usage in ExecuteScript Processor

ession due to org.luaj.vm2.LuaError: script:98 vm error: java.lang.ClassNotFoundException: org.apache.commons.io.IOUtils: org.luaj.vm2.LuaError: script:98 vm error: java.lang.ClassNotFoundException: org.apache.commons.io.IOUtils Any help here? On Thu, Apr 21, 2016 at 10:58 AM, Madhukar Thota &l

Re: ExecuteScript Processor Performance

concurrent tasks using executescript processor. -Madhu On Mon, May 2, 2016 at 11:55 AM, Madhukar Thota <madhukar.th...@gmail.com> wrote: > Thanks Matt and Joe for your input. I will go through your suggestions. > > On Mon, May 2, 2016 at 10:16 AM, Matt Burgess <mattyb...@gmail.com&

Re: ExecuteScript Processor Performance

p in Java. Looks like it might > > be a straightforward processor so now that you've gotten the precise > > logic you want you can turn it into a full nifi processor and you'll > > get solid performance. > > > > Thanks > > Joe > > > > On Mon, May 2, 20

ExecuteScript Processor Performance

Hi I am using ExecuteScript Processor( using python/jython script pasted below) to process http querystring along with useragent parsing. The processor is very slow and not able to handle heavy load. Lot of them of getting queued and waiting for the processor to process it. How can i improve the

Re: Nifi + opentsdb

ml >> >> >> Joe >> - - - - - - >> Joseph Percivall >> linkedin.com/in/Percivall >> e: joeperciv...@yahoo.com >> >> >> >> >> On Monday, April 25, 2016 10:46 PM, Joe Witt <joe.w...@gmail.com> wrote: >> Madhu, >>

Nifi + opentsdb

Friends, Just checking to see if anyone in the community using Nifi or custom Nifi processor to write the data into opentsdb? Any input is appreciated. -Madhu

Re: Lua usage in ExecuteScript Processor

, 2016 at 5:21 PM, Madhukar Thota <madhukar.th...@gmail.com> wrote: > I am trying to read the lua file this way, but its not working. How to > read the lua files from module directory and use it in execution? > > luajava.LuaState = luajava.LuaStateFactory.newLuaState() > > >

Lua usage in ExecuteScript Processor

Friends, Can anyone share an sample example on how to use Lua in ExecuteScript Processor? We have bunch of lua scripts which we would like to use for data processing. Any help is appreciated. Thanks Madhu

Re: Kafka Schema registry

pport. Can you point us to what you are looking at > specifically? > > Thanks > Joe > > On Wed, Apr 13, 2016 at 1:34 PM, Madhukar Thota > <madhukar.th...@gmail.com> wrote: > > Friends, > > > > Is it possible to use Schema registry with Kafka Processors to store and > > retrive Avro schema? > > > > -Madhu >

Kafka Schema registry

Friends, Is it possible to use Schema registry with Kafka Processors to store and retrive Avro schema? -Madhu

remote zookeeper cluster with Nifi

We have a dedicated zookeeper cluster which we would like to use with Nifi for state management. Is it possible to configure remote zookeeper cluster instead of embedded zookeepr?

Re: ExecuteSQL to elasticsearch

g here or do i need extra processor to convert into right format what elasticsearch understands? On Thu, Apr 7, 2016 at 7:49 AM, Madhukar Thota <madhukar.th...@gmail.com> wrote: > Friends, > > I am exploring ExecuteSQL processor in nifi and my goal to get sql data > ingested i

Re: nifi processor to parse+update the current json on the fly

Here is an example of json to json conversion using Groovy with JsonSlurper. http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html On Thu, Apr 7, 2016 at 11:31 AM, Thad Guidry wrote: > Philippe, > > I would encourage you to just use Groovy with

Re: problem with putElasticsearch processor

I think Elasticsearch processor uses transport client not http. So you should use 9300(transport port) port not 9200(http port). On Wed, Apr 6, 2016 at 12:41 PM, wrote: > Hello > > My context nifi 0.6.0 on Ubuntu 14.0 > > my small use case is : > > ‘sending JSON

ExecuteSQL to elasticsearch

Friends, I am exploring ExecuteSQL processor in nifi and my goal to get sql data ingested in Elasticsearch. Can someone share or guide what's the flow looks like? Thanks in Advance.

Re: How to add python modules ?

i made small progess but seeing different execption not sure why i am seeing nil value. error: 22:58:35 EDT ERROR 6f15a6f2-7744-404c-9961-f545d3f29042 ExecuteScript[id=6f15a6f2-7744-404c-9961-f545d3f29042] Failed to process session due to org.apache.nifi.processor.exception.ProcessException:

Re: How to add python modules ?

nary from the main script and pass it into the > callback constructor. Then process() can update it, and you can use the > populated dictionary after process() returns to set attributes and such. > > Regards, > Matt > > > On Mar 30, 2016, at 5:00 PM, Madhukar Thota <madhu

Re: How to add python modules ?

Matt, I tired the following code but i am getting the following error. Can you help me where i am doing wrong? Error: 16:56:10 EDT ERROR 6f15a6f2-7744-404c-9961-f545d3f29042 ExecuteScript[id=6f15a6f2-7744-404c-9961-f545d3f29042] Failed to process session due to

Re: String conversion to Int, float double

e fine. > > Thanks > Joe > > On Wed, Mar 30, 2016 at 11:16 AM, Madhukar Thota > <madhukar.th...@gmail.com> wrote: > > I was able to construct the Json with right data type output from > > ExecuteScript and sending to Kafka directly. The problem i am seeing i

Re: String conversion to Int, float double

ied this by the way, but the type > conversion certainly works for the final attributes in the Avro documents. > Conrad > > From: Madhukar Thota <madhukar.th...@gmail.com> > Reply-To: "users@nifi.apache.org" <users@nifi.apache.org> > Date: Friday, 25 March 2016

Re: String conversion to Int, float double

> > but end with have ".type" ? > > > > So in your example there would be: > > > > rt_firstbyte ${http.param.t_resp} > > rt_firstbyte.type int > > > > If it doesn't find a type attribute then it defaults to String as it does > > today. > &

Re: String conversion to Int, float double

do > with a given attribute that is of type Int/Long/etc..? > > Thanks > Joe > > > > On Fri, Mar 25, 2016 at 8:01 AM, Chase Cunningham <ch...@thecynja.com> > wrote: > > unsubscribe > > > > > > On 3/25/16 9:01 AM, Madhukar Thota wrote: > >

Re: String conversion to Int, float double

other than Strings. > > -Bryan > > > On Thu, Mar 24, 2016 at 3:30 PM, Madhukar Thota <madhukar.th...@gmail.com> > wrote: > >> Hi i am trying to convert string value to integer in UpdateAtrributes >> using toNumber like this >> >> >> ${http.param.t_re

String conversion to Int, float double

Hi i am trying to convert string value to integer in UpdateAtrributes using toNumber like this ${http.param.t_resp:toNumber()} where http.param.t_resp = "132" but when the fileattribute pushed to Attributetojson processor , i am stilling seeing it as string. Am i am doing something wrong? and

Re: How to add python modules ?

=372_done=393_other=t_domloaded%7C364=http%3A%2F%2Flocalhost%3A63342%2FBeacon%2Ftest.html==http%3A%2F%2Flocalhost%3A63342%2FBeacon%2Ftest.html=0.9& vis.st=visible -Madhu On Thu, Mar 24, 2016 at 12:34 AM, Madhukar Thota <madhukar.th...@gmail.com> wrote: > Hi Matt, > > Thank y

Re: How to add python modules ?

> > I am new to Python as well, but am happy to help if I can with any issues > you run into, as it will help me learn more as well :) > > Regards, > Matt > > > On Thu, Mar 24, 2016 at 12:10 AM, Madhukar Thota <madhukar.th...@gmail.com > > wrote: > >&g

Re: How to add python modules ?

fields. I am kind of new to python and also this my first attempt of using python with nifi. Any help is appreciated. On Wed, Mar 23, 2016 at 11:31 PM, Madhukar Thota <madhukar.th...@gmail.com> wrote: > Hi > > I am trying to use the following script to parse http.he

How to add python modules ?

Hi I am trying to use the following script to parse http.headers.useragent with python useragent module using ExecuteScript Processor. Script: #!/usr/bin/env python2.7 from user_agents import parse flowFile = session.get() if (flowFile != None): flowFile = session.putAttribute(flowFile,

Execute script - python example

Hi I am looking for an example in python to convert a new field based on attribute value. Let say syslog.facilty holds value 23, based on the value i want to create new field with text value like syslog.facility_label=LOCAL7 If this transformation possible with existing processors, please

Re: Log4j/logback parser via syslog

> There are some cool blogs about them here: > > http://funnifi.blogspot.com/2016/02/executescript-processor-hello-world.html > > -Bryan > > On Fri, Feb 12, 2016 at 10:48 AM, Madhukar Thota <madhukar.th...@gmail.com > > wrote: > >> Thanks Bryan. I will look