Re: Nifi ExecuteScript slow performance
Vyshali, Another trick you can do that works very well if you have a lot of flowfiles to process is use session.get(int) to grab a batch. Obviously, keep the volume you grab tuned to reasonable memory limits and all that, but you can use that to make the script do a lot more work in one single run. I have a flow that ends up processing millions of JSON files, and grabbing 500 of them at once and merging them into a JSON array to create a record batch really helps. On Mon, Nov 13, 2017 at 12:23 PM, Vyshaliwrote: > Thank you so much Matt. > I will try the solutions provided and come back in case of questions. > > Thanks, > Vyshali > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ >
Re: Nifi ExecuteScript slow performance
Thank you so much Matt. I will try the solutions provided and come back in case of questions. Thanks, Vyshali -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Nifi ExecuteScript slow performance
Hi Matt, I'm using Jython in executescript because of my requirement.I cant switch to groovy because I'm using packages supported by Python.Is there any way to increase the speed of the executescript processor.Please help me with your ideas. Thanks, Vyshali -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Nifi ExecuteScript slow performance
Thanks for the reply Joe, As matt suggested we moved to groovy. Thanks Bala -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735p13782.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Nifi ExecuteScript slow performance
Thanks for the reply matt, Jython is slow, As you suggested we moved to groovy. ExecuteScript is taking more than 2 seconds to execute below high level logic The logic: 1) Split data from flowfile 2) Query influxdb with splitted data (influxdb jar file used/external dependency) 3) write to influxdb 4) write flow file attributes Without influxdb calling, the code is fast. Can you suggest me about improving the performance of the below mentioned groovy script Thanks Bala -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735p13781.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Nifi ExecuteScript slow performance
Bala, Jython in ExecuteScript is noticably slower than other languages like Javascript and Groovy, but it shouldn't be that slow. Can you share your script code? Also, is this just to investigate before writing a more complex script in ExecuteScript, or do you just want to add an attribute to a flowfile? If the latter, UpdateAttribute can do that, but I suspect there is more to it than that (custom logic, e.g.). Regards, Matt On Thu, Oct 27, 2016 at 8:08 AM, Joe Witt <joe.w...@gmail.com> wrote: > Bala, > > Are you saying that step 2 (executing the script) is taking three > seconds? Is that per message? Can you show the log or screenshot of > how you're tracking that? > > Thanks > Joe > > On Thu, Oct 27, 2016 at 6:02 AM, balacode63 <balamurugan.c...@gmail.com> > wrote: >> Dear All, >> >> I've added a ExecuteScript in python. This script is a simple script which >> will just add one attribute to the flowfile. The flow is definded as below, >> >>1. Listen from mqtt (ConsumeMQTT) >>2. Add attribute to nifi flow (Execute script) >>3. Write to a file/post/ or some custom logic >> >> But its taking 3 seconds to process. Please guide me. >> >> Thanks, >> Bala >> >> >> >> -- >> View this message in context: >> http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html >> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Nifi ExecuteScript slow performance
Bala, Are you saying that step 2 (executing the script) is taking three seconds? Is that per message? Can you show the log or screenshot of how you're tracking that? Thanks Joe On Thu, Oct 27, 2016 at 6:02 AM, balacode63 <balamurugan.c...@gmail.com> wrote: > Dear All, > > I've added a ExecuteScript in python. This script is a simple script which > will just add one attribute to the flowfile. The flow is definded as below, > >1. Listen from mqtt (ConsumeMQTT) >2. Add attribute to nifi flow (Execute script) >3. Write to a file/post/ or some custom logic > > But its taking 3 seconds to process. Please guide me. > > Thanks, > Bala > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html > Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Nifi ExecuteScript slow performance
Dear All, I've added a ExecuteScript in python. This script is a simple script which will just add one attribute to the flowfile. The flow is definded as below, 1. Listen from mqtt (ConsumeMQTT) 2. Add attribute to nifi flow (Execute script) 3. Write to a file/post/ or some custom logic But its taking 3 seconds to process. Please guide me. Thanks, Bala -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.