Re: Requesting PYTHON documentation for the Python side of a NiFi ExecuteScript processor execution

[Matt Burgess]

Paul,

The first thing I would mention (which helps explain the rest) is that
NiFi uses the Jython [1] engine, it is not pure Python. So although
you cannot use native (CPython) libraries like Pandas, you can import
and use Java classes, which is what the NiFi API is written in. So
when you say "Python functions available for the ExecuteScript API",
what that really means is that your Jython code can call Java methods
on Java objects, such as outputstream.write() or session.transfer().

A good place to start to understand the NiFi API is the Developer
Guide [2]. This covers the concepts you may run into in your script,
in fact way more than you may need. To narrow it down, start with the
ProcessSession API [3]. This shows you the methods you can call on the
session, such as writing to a flow file using a StreamCallback [4],
and within that you can see the process method [5].  For the
ProcessSession, you can also see the transfer() and putAttribute()
methods which are very often used in scripts. There are other places
where you might interact with the API, such as FlowFile [6].

I have a few examples on my blog [7] of using Jython in ExecuteScript,
such as converting JSON into a different JSON structure [8]. I tried
to explain the usage of the NiFi API there though I admit it could be
more in-depth. In addition I plan to write a good amount of
documentation (to include language-specific examples) under NIFI-1954
[9]. I am the author of the ExecuteScript Cookbook series you
mentioned as well. Please feel free to respond with any additional
questions you may have, happy to help.

Regards,
Matt

[1] https://www.jython.org/
[2] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
[3] 
https://javadoc.io/static/org.apache.nifi/nifi-api/1.16.1/org/apache/nifi/processor/ProcessSession.html
[4] 
https://javadoc.io/static/org.apache.nifi/nifi-api/1.16.1/org/apache/nifi/processor/ProcessSession.html#write-org.apache.nifi.flowfile.FlowFile-org.apache.nifi.processor.io.StreamCallback-
[5] 
https://javadoc.io/static/org.apache.nifi/nifi-api/1.16.1/org/apache/nifi/processor/io/StreamCallback.html
[6] 
https://javadoc.io/static/org.apache.nifi/nifi-api/1.16.1/org/apache/nifi/flowfile/FlowFile.html
[7] https://funnifi.blogspot.com/
[8] 
https://funnifi.blogspot.com/2016/03/executescript-json-to-json-revisited_14.html
[9] https://issues.apache.org/jira/browse/NIFI-1954

On Wed, May 4, 2022 at 1:07 PM Markiewicz, Paul  wrote:
>
> Hello, my name is PaulM.
>
> I am very new at both NiFi and relatively new at Python.
>
>
> I have successfully cobbled together a NiFi ExecuteScript processor
> (complete with embeded Python Code).
> That python code CAN/DOES read the original Flow-File content, sets/updates
> a NiFi attribute of my choosing,
> repopulates the Flow-File content with some junk (io.environ) data (and some
> other Hello-World lines..).
> Then successfully, returns (from the Python execution), the new attribute
> and new Flow-File content to
> the NiFi ExecuteScript.
>
>
> I had this success by reading a PART-1/PART-2 example (of Python) found
> within a cloudera link..
> (((
> https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-
> part-1/ta-p/248922 ))).
>
>
> What I am hoping to get from this developer mailing list eMail request (((
> dev@nifi.apache.org )))
> is where/how to locate DOCUMENTATION for what is available within the
> **PYTHON SIDE** of these library calls.
>
>
> For example, my Python uses these IMPORT STATEMENTS:
>
>
> import os
> import datetime
>
> from org.python.core.util.FileUtil import wrap
> from org.apache.nifi.processors.script import ExecuteScript
> from org.apache.commons.io import IOUtils
> from java.nio.charset import StandardCharsets
> from org.apache.nifi.processor.io import StreamCallback
>
>
> ...and from the example code (which does work...)  I am especially
> interested in
> the PYTHON documentation for "ExecuteScript" so that I can also know things
> like the complement
> FAILURE variable name, to this SUCCESS:
>
> session.transfer(flow_file, ExecuteScript.REL_SUCCESS)
>
>
> ... and learn all the other Python functions available for the ExecuteScript
> API.
>
>
> Hopefully that documentation would ALSO point out important concepts like
> inside my "class PyStreamCallback(StreamCallback):" the execution flow will
> automagically call "process"
> within that class... and all the various (correct) data types that are
> expected within the Python execution.
>
> ... and that the
> "outputStream.write(bytearray(newFlowContentStr.encode('utf-8')))"
> expects a parameter of type bytearray, to the write() function.
>
>
>
> I thank you in advance.
>
> Sincerely,
> PaulM
>
>

Requesting PYTHON documentation for the Python side of a NiFi ExecuteScript processor execution

[Markiewicz, Paul]

Hello, my name is PaulM.

I am very new at both NiFi and relatively new at Python.


I have successfully cobbled together a NiFi ExecuteScript processor
(complete with embeded Python Code).
That python code CAN/DOES read the original Flow-File content, sets/updates
a NiFi attribute of my choosing,
repopulates the Flow-File content with some junk (io.environ) data (and some
other Hello-World lines..).
Then successfully, returns (from the Python execution), the new attribute
and new Flow-File content to 
the NiFi ExecuteScript.


I had this success by reading a PART-1/PART-2 example (of Python) found
within a cloudera link..
(((
https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-
part-1/ta-p/248922 ))).


What I am hoping to get from this developer mailing list eMail request (((
dev@nifi.apache.org )))
is where/how to locate DOCUMENTATION for what is available within the
**PYTHON SIDE** of these library calls.


For example, my Python uses these IMPORT STATEMENTS:


import os
import datetime

from org.python.core.util.FileUtil import wrap
from org.apache.nifi.processors.script import ExecuteScript
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback


...and from the example code (which does work...)  I am especially
interested in 
the PYTHON documentation for "ExecuteScript" so that I can also know things
like the complement 
FAILURE variable name, to this SUCCESS:

session.transfer(flow_file, ExecuteScript.REL_SUCCESS)


... and learn all the other Python functions available for the ExecuteScript
API.


Hopefully that documentation would ALSO point out important concepts like 
inside my "class PyStreamCallback(StreamCallback):" the execution flow will
automagically call "process"
within that class... and all the various (correct) data types that are
expected within the Python execution.

... and that the
"outputStream.write(bytearray(newFlowContentStr.encode('utf-8')))"
expects a parameter of type bytearray, to the write() function.



I thank you in advance.

Sincerely,
PaulM




smime.p7s
Description: S/MIME cryptographic signature

NiFi - ExecuteScript

[Секлецов Даниил Иванович]

Hello!

Please, can you tell me how to debug code in ExecuteScript blocks?
I'm using ExecuteScript 1.12.1, ScriptEngine: ECMAScript.


УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: Это электронное сообщение и любые документы, 
приложенные к нему, содержат конфиденциальную информацию. Настоящим уведомляем 
Вас о том, что если это сообщение не предназначено Вам, использование, 
копирование, распространение информации, содержащейся в настоящем сообщении, а 
также осуществление любых действий на основе этой информации, строго запрещено. 
Если Вы получили это сообщение по ошибке, пожалуйста, сообщите об этом 
отправителю по электронной почте и удалите это сообщение. CONFIDENTIALITY 
NOTICE: This email and any files attached to it are confidential. If you are 
not the intended recipient you are notified that using, copying, distributing 
or taking any action in reliance on the contents of this information is 
strictly prohibited. If you have received this email in error please notify the 
sender and delete this email.

Re: Nifi ExecuteScript slow performance

[Mike Thomsen]

Vyshali,

Another trick you can do that works very well if you have a lot of
flowfiles to process is use session.get(int) to grab a batch. Obviously,
keep the volume you grab tuned to reasonable memory limits and all that,
but you can use that to make the script do a lot more work in one single
run. I have a flow that ends up processing millions of JSON files, and
grabbing 500 of them at once and merging them into a JSON array to create a
record batch really helps.

On Mon, Nov 13, 2017 at 12:23 PM, Vyshali  wrote:

> Thank you so much Matt.
> I will try the solutions provided and come back in case of questions.
>
> Thanks,
> Vyshali
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: Nifi ExecuteScript slow performance

[Vyshali]

Thank you so much Matt.
I will try the solutions provided and come back in case of questions.

Thanks,
Vyshali



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Nifi ExecuteScript slow performance

[Vyshali]

Hi Matt,

I'm using Jython in executescript because of my requirement.I cant switch to
groovy because I'm using packages supported by Python.Is there any way to
increase the speed of the executescript processor.Please help me with your
ideas.

Thanks,
Vyshali



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Nifi ExecuteScript slow performance

[balacode63]

Thanks for the reply Joe,

As matt suggested we moved to groovy. 

Thanks
Bala



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735p13782.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Nifi ExecuteScript slow performance

[balacode63]

Thanks for the reply matt, 
Jython is slow, As you suggested we moved to groovy. ExecuteScript is taking
more than 2 seconds to execute below high level logic

The logic: 
1) Split data from flowfile
2) Query influxdb with splitted data (influxdb jar file used/external
dependency)
3) write to influxdb
4) write flow file attributes

Without influxdb calling, the code is fast. Can you suggest me about
improving the performance of the below mentioned groovy script



Thanks
Bala




--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735p13781.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Nifi ExecuteScript slow performance

[Matt Burgess]

Bala,

Jython in ExecuteScript is noticably slower than other languages like
Javascript and Groovy, but it shouldn't be that slow. Can you share
your script code? Also, is this just to investigate before writing a
more complex script in ExecuteScript, or do you just want to add an
attribute to a flowfile? If the latter, UpdateAttribute can do that,
but I suspect there is more to it than that (custom logic, e.g.).

Regards,
Matt

On Thu, Oct 27, 2016 at 8:08 AM, Joe Witt <joe.w...@gmail.com> wrote:
> Bala,
>
> Are you saying that step 2 (executing the script) is taking three
> seconds?  Is that per message?  Can you show the log or screenshot of
> how you're tracking that?
>
> Thanks
> Joe
>
> On Thu, Oct 27, 2016 at 6:02 AM, balacode63 <balamurugan.c...@gmail.com> 
> wrote:
>> Dear All,
>>
>> I've added a ExecuteScript in python. This script is a simple script which
>> will just add one attribute to the flowfile. The flow is definded as below,
>>
>>1. Listen from mqtt (ConsumeMQTT)
>>2. Add attribute to nifi flow (Execute script)
>>3. Write to a file/post/ or some custom logic
>>
>> But its taking 3 seconds to process. Please guide me.
>>
>> Thanks,
>> Bala
>>
>>
>>
>> --
>> View this message in context: 
>> http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html
>> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Nifi ExecuteScript slow performance

[Joe Witt]

Bala,

Are you saying that step 2 (executing the script) is taking three
seconds?  Is that per message?  Can you show the log or screenshot of
how you're tracking that?

Thanks
Joe

On Thu, Oct 27, 2016 at 6:02 AM, balacode63 <balamurugan.c...@gmail.com> wrote:
> Dear All,
>
> I've added a ExecuteScript in python. This script is a simple script which
> will just add one attribute to the flowfile. The flow is definded as below,
>
>1. Listen from mqtt (ConsumeMQTT)
>2. Add attribute to nifi flow (Execute script)
>3. Write to a file/post/ or some custom logic
>
> But its taking 3 seconds to process. Please guide me.
>
> Thanks,
> Bala
>
>
>
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Nifi ExecuteScript slow performance

[balacode63]

Dear All,

I've added a ExecuteScript in python. This script is a simple script which
will just add one attribute to the flowfile. The flow is definded as below,

   1. Listen from mqtt (ConsumeMQTT)
   2. Add attribute to nifi flow (Execute script)
   3. Write to a file/post/ or some custom logic

But its taking 3 seconds to process. Please guide me.

Thanks,
Bala



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-ExecuteScript-slow-performance-tp13735.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.