Re: Requesting PYTHON documentation for the Python side of a NiFi ExecuteScript processor execution
Paul, The first thing I would mention (which helps explain the rest) is that NiFi uses the Jython [1] engine, it is not pure Python. So although you cannot use native (CPython) libraries like Pandas, you can import and use Java classes, which is what the NiFi API is written in. So when you say "Python functions available for the ExecuteScript API", what that really means is that your Jython code can call Java methods on Java objects, such as outputstream.write() or session.transfer(). A good place to start to understand the NiFi API is the Developer Guide [2]. This covers the concepts you may run into in your script, in fact way more than you may need. To narrow it down, start with the ProcessSession API [3]. This shows you the methods you can call on the session, such as writing to a flow file using a StreamCallback [4], and within that you can see the process method [5]. For the ProcessSession, you can also see the transfer() and putAttribute() methods which are very often used in scripts. There are other places where you might interact with the API, such as FlowFile [6]. I have a few examples on my blog [7] of using Jython in ExecuteScript, such as converting JSON into a different JSON structure [8]. I tried to explain the usage of the NiFi API there though I admit it could be more in-depth. In addition I plan to write a good amount of documentation (to include language-specific examples) under NIFI-1954 [9]. I am the author of the ExecuteScript Cookbook series you mentioned as well. Please feel free to respond with any additional questions you may have, happy to help. Regards, Matt [1] https://www.jython.org/ [2] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html [3] https://javadoc.io/static/org.apache.nifi/nifi-api/1.16.1/org/apache/nifi/processor/ProcessSession.html [4] https://javadoc.io/static/org.apache.nifi/nifi-api/1.16.1/org/apache/nifi/processor/ProcessSession.html#write-org.apache.nifi.flowfile.FlowFile-org.apache.nifi.processor.io.StreamCallback- [5] https://javadoc.io/static/org.apache.nifi/nifi-api/1.16.1/org/apache/nifi/processor/io/StreamCallback.html [6] https://javadoc.io/static/org.apache.nifi/nifi-api/1.16.1/org/apache/nifi/flowfile/FlowFile.html [7] https://funnifi.blogspot.com/ [8] https://funnifi.blogspot.com/2016/03/executescript-json-to-json-revisited_14.html [9] https://issues.apache.org/jira/browse/NIFI-1954 On Wed, May 4, 2022 at 1:07 PM Markiewicz, Paul wrote: > > Hello, my name is PaulM. > > I am very new at both NiFi and relatively new at Python. > > > I have successfully cobbled together a NiFi ExecuteScript processor > (complete with embeded Python Code). > That python code CAN/DOES read the original Flow-File content, sets/updates > a NiFi attribute of my choosing, > repopulates the Flow-File content with some junk (io.environ) data (and some > other Hello-World lines..). > Then successfully, returns (from the Python execution), the new attribute > and new Flow-File content to > the NiFi ExecuteScript. > > > I had this success by reading a PART-1/PART-2 example (of Python) found > within a cloudera link.. > ((( > https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook- > part-1/ta-p/248922 ))). > > > What I am hoping to get from this developer mailing list eMail request ((( > dev@nifi.apache.org ))) > is where/how to locate DOCUMENTATION for what is available within the > **PYTHON SIDE** of these library calls. > > > For example, my Python uses these IMPORT STATEMENTS: > > > import os > import datetime > > from org.python.core.util.FileUtil import wrap > from org.apache.nifi.processors.script import ExecuteScript > from org.apache.commons.io import IOUtils > from java.nio.charset import StandardCharsets > from org.apache.nifi.processor.io import StreamCallback > > > ...and from the example code (which does work...) I am especially > interested in > the PYTHON documentation for "ExecuteScript" so that I can also know things > like the complement > FAILURE variable name, to this SUCCESS: > > session.transfer(flow_file, ExecuteScript.REL_SUCCESS) > > > ... and learn all the other Python functions available for the ExecuteScript > API. > > > Hopefully that documentation would ALSO point out important concepts like > inside my "class PyStreamCallback(StreamCallback):" the execution flow will > automagically call "process" > within that class... and all the various (correct) data types that are > expected within the Python execution. > > ... and that the > "outputStream.write(bytearray(newFlowContentStr.encode('utf-8')))" > expects a parameter of type bytearray, to the write() function. > > > > I thank you in advance. > > Sincerely, > PaulM > >
Requesting PYTHON documentation for the Python side of a NiFi ExecuteScript processor execution
Hello, my name is PaulM. I am very new at both NiFi and relatively new at Python. I have successfully cobbled together a NiFi ExecuteScript processor (complete with embeded Python Code). That python code CAN/DOES read the original Flow-File content, sets/updates a NiFi attribute of my choosing, repopulates the Flow-File content with some junk (io.environ) data (and some other Hello-World lines..). Then successfully, returns (from the Python execution), the new attribute and new Flow-File content to the NiFi ExecuteScript. I had this success by reading a PART-1/PART-2 example (of Python) found within a cloudera link.. ((( https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook- part-1/ta-p/248922 ))). What I am hoping to get from this developer mailing list eMail request ((( dev@nifi.apache.org ))) is where/how to locate DOCUMENTATION for what is available within the **PYTHON SIDE** of these library calls. For example, my Python uses these IMPORT STATEMENTS: import os import datetime from org.python.core.util.FileUtil import wrap from org.apache.nifi.processors.script import ExecuteScript from org.apache.commons.io import IOUtils from java.nio.charset import StandardCharsets from org.apache.nifi.processor.io import StreamCallback ...and from the example code (which does work...) I am especially interested in the PYTHON documentation for "ExecuteScript" so that I can also know things like the complement FAILURE variable name, to this SUCCESS: session.transfer(flow_file, ExecuteScript.REL_SUCCESS) ... and learn all the other Python functions available for the ExecuteScript API. Hopefully that documentation would ALSO point out important concepts like inside my "class PyStreamCallback(StreamCallback):" the execution flow will automagically call "process" within that class... and all the various (correct) data types that are expected within the Python execution. ... and that the "outputStream.write(bytearray(newFlowContentStr.encode('utf-8')))" expects a parameter of type bytearray, to the write() function. I thank you in advance. Sincerely, PaulM smime.p7s Description: S/MIME cryptographic signature
NiFi - ExecuteScript
Hello! Please, can you tell me how to debug code in ExecuteScript blocks? I'm using ExecuteScript 1.12.1, ScriptEngine: ECMAScript. УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: Это электронное сообщение и любые документы, приложенные к нему, содержат конфиденциальную информацию. Настоящим уведомляем Вас о том, что если это сообщение не предназначено Вам, использование, копирование, распространение информации, содержащейся в настоящем сообщении, а также осуществление любых действий на основе этой информации, строго запрещено. Если Вы получили это сообщение по ошибке, пожалуйста, сообщите об этом отправителю по электронной почте и удалите это сообщение. CONFIDENTIALITY NOTICE: This email and any files attached to it are confidential. If you are not the intended recipient you are notified that using, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify the sender and delete this email.
Re: Nifi ExecuteScript slow performance
Vyshali, Another trick you can do that works very well if you have a lot of flowfiles to process is use session.get(int) to grab a batch. Obviously, keep the volume you grab tuned to reasonable memory limits and all that, but you can use that to make the script do a lot more work in one single run. I have a flow that ends up processing millions of JSON files, and grabbing 500 of them at once and merging them into a JSON array to create a record batch really helps. On Mon, Nov 13, 2017 at 12:23 PM, Vyshaliwrote: > Thank you so much Matt. > I will try the solutions provided and come back in case of questions. > > Thanks, > Vyshali > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ >
Re: Nifi ExecuteScript slow performance
Thank you so much Matt. I will try the solutions provided and come back in case of questions. Thanks, Vyshali -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Nifi ExecuteScript slow performance
Hi Matt, I'm using Jython in executescript because of my requirement.I cant switch to groovy because I'm using packages supported by Python.Is there any way to increase the speed of the executescript processor.Please help me with your ideas. Thanks, Vyshali -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Nifi ExecuteScript slow performance
Thanks for the reply Joe, As matt suggested we moved to groovy. Thanks Bala -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735p13782.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Nifi ExecuteScript slow performance
Thanks for the reply matt, Jython is slow, As you suggested we moved to groovy. ExecuteScript is taking more than 2 seconds to execute below high level logic The logic: 1) Split data from flowfile 2) Query influxdb with splitted data (influxdb jar file used/external dependency) 3) write to influxdb 4) write flow file attributes Without influxdb calling, the code is fast. Can you suggest me about improving the performance of the below mentioned groovy script Thanks Bala -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735p13781.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Nifi ExecuteScript slow performance
Bala, Jython in ExecuteScript is noticably slower than other languages like Javascript and Groovy, but it shouldn't be that slow. Can you share your script code? Also, is this just to investigate before writing a more complex script in ExecuteScript, or do you just want to add an attribute to a flowfile? If the latter, UpdateAttribute can do that, but I suspect there is more to it than that (custom logic, e.g.). Regards, Matt On Thu, Oct 27, 2016 at 8:08 AM, Joe Witt <joe.w...@gmail.com> wrote: > Bala, > > Are you saying that step 2 (executing the script) is taking three > seconds? Is that per message? Can you show the log or screenshot of > how you're tracking that? > > Thanks > Joe > > On Thu, Oct 27, 2016 at 6:02 AM, balacode63 <balamurugan.c...@gmail.com> > wrote: >> Dear All, >> >> I've added a ExecuteScript in python. This script is a simple script which >> will just add one attribute to the flowfile. The flow is definded as below, >> >>1. Listen from mqtt (ConsumeMQTT) >>2. Add attribute to nifi flow (Execute script) >>3. Write to a file/post/ or some custom logic >> >> But its taking 3 seconds to process. Please guide me. >> >> Thanks, >> Bala >> >> >> >> -- >> View this message in context: >> http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html >> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Nifi ExecuteScript slow performance
Bala, Are you saying that step 2 (executing the script) is taking three seconds? Is that per message? Can you show the log or screenshot of how you're tracking that? Thanks Joe On Thu, Oct 27, 2016 at 6:02 AM, balacode63 <balamurugan.c...@gmail.com> wrote: > Dear All, > > I've added a ExecuteScript in python. This script is a simple script which > will just add one attribute to the flowfile. The flow is definded as below, > >1. Listen from mqtt (ConsumeMQTT) >2. Add attribute to nifi flow (Execute script) >3. Write to a file/post/ or some custom logic > > But its taking 3 seconds to process. Please guide me. > > Thanks, > Bala > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html > Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Nifi ExecuteScript slow performance
Dear All, I've added a ExecuteScript in python. This script is a simple script which will just add one attribute to the flowfile. The flow is definded as below, 1. Listen from mqtt (ConsumeMQTT) 2. Add attribute to nifi flow (Execute script) 3. Write to a file/post/ or some custom logic But its taking 3 seconds to process. Please guide me. Thanks, Bala -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.