Re: Nifi ExecuteScript slow performance

[Mike Thomsen]

Vyshali,

Another trick you can do that works very well if you have a lot of
flowfiles to process is use session.get(int) to grab a batch. Obviously,
keep the volume you grab tuned to reasonable memory limits and all that,
but you can use that to make the script do a lot more work in one single
run. I have a flow that ends up processing millions of JSON files, and
grabbing 500 of them at once and merging them into a JSON array to create a
record batch really helps.

On Mon, Nov 13, 2017 at 12:23 PM, Vyshali  wrote:

> Thank you so much Matt.
> I will try the solutions provided and come back in case of questions.
>
> Thanks,
> Vyshali
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: Nifi ExecuteScript slow performance

[Vyshali]

Thank you so much Matt.
I will try the solutions provided and come back in case of questions.

Thanks,
Vyshali



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Nifi ExecuteScript slow performance

[Vyshali]

Hi Matt,

I'm using Jython in executescript because of my requirement.I cant switch to
groovy because I'm using packages supported by Python.Is there any way to
increase the speed of the executescript processor.Please help me with your
ideas.

Thanks,
Vyshali



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Nifi ExecuteScript slow performance

[balacode63]

Thanks for the reply Joe,

As matt suggested we moved to groovy. 

Thanks
Bala



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735p13782.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Nifi ExecuteScript slow performance

[balacode63]

Thanks for the reply matt, 
Jython is slow, As you suggested we moved to groovy. ExecuteScript is taking
more than 2 seconds to execute below high level logic

The logic: 
1) Split data from flowfile
2) Query influxdb with splitted data (influxdb jar file used/external
dependency)
3) write to influxdb
4) write flow file attributes

Without influxdb calling, the code is fast. Can you suggest me about
improving the performance of the below mentioned groovy script



Thanks
Bala




--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735p13781.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Nifi ExecuteScript slow performance

[Matt Burgess]

Bala,

Jython in ExecuteScript is noticably slower than other languages like
Javascript and Groovy, but it shouldn't be that slow. Can you share
your script code? Also, is this just to investigate before writing a
more complex script in ExecuteScript, or do you just want to add an
attribute to a flowfile? If the latter, UpdateAttribute can do that,
but I suspect there is more to it than that (custom logic, e.g.).

Regards,
Matt

On Thu, Oct 27, 2016 at 8:08 AM, Joe Witt <joe.w...@gmail.com> wrote:
> Bala,
>
> Are you saying that step 2 (executing the script) is taking three
> seconds?  Is that per message?  Can you show the log or screenshot of
> how you're tracking that?
>
> Thanks
> Joe
>
> On Thu, Oct 27, 2016 at 6:02 AM, balacode63 <balamurugan.c...@gmail.com> 
> wrote:
>> Dear All,
>>
>> I've added a ExecuteScript in python. This script is a simple script which
>> will just add one attribute to the flowfile. The flow is definded as below,
>>
>>1. Listen from mqtt (ConsumeMQTT)
>>2. Add attribute to nifi flow (Execute script)
>>3. Write to a file/post/ or some custom logic
>>
>> But its taking 3 seconds to process. Please guide me.
>>
>> Thanks,
>> Bala
>>
>>
>>
>> --
>> View this message in context: 
>> http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html
>> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Nifi ExecuteScript slow performance

[Joe Witt]

Bala,

Are you saying that step 2 (executing the script) is taking three
seconds?  Is that per message?  Can you show the log or screenshot of
how you're tracking that?

Thanks
Joe

On Thu, Oct 27, 2016 at 6:02 AM, balacode63 <balamurugan.c...@gmail.com> wrote:
> Dear All,
>
> I've added a ExecuteScript in python. This script is a simple script which
> will just add one attribute to the flowfile. The flow is definded as below,
>
>1. Listen from mqtt (ConsumeMQTT)
>2. Add attribute to nifi flow (Execute script)
>3. Write to a file/post/ or some custom logic
>
> But its taking 3 seconds to process. Please guide me.
>
> Thanks,
> Bala
>
>
>
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Nifi ExecuteScript slow performance

[balacode63]

Dear All,

I've added a ExecuteScript in python. This script is a simple script which
will just add one attribute to the flowfile. The flow is definded as below,

   1. Listen from mqtt (ConsumeMQTT)
   2. Add attribute to nifi flow (Execute script)
   3. Write to a file/post/ or some custom logic

But its taking 3 seconds to process. Please guide me.

Thanks,
Bala



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.