In general, I would recommend moving the Python code to Flask-based rest services running on something like Kubernetes. That way you can access them with InvokeHttp which has a much lower overhead than ExecuteStreamCommand (no CPython startup per run). Creating containerized Python services is not hard, so I would recommend that route as a way of really dropping the resource needs of your NiFi servers.
On Wed, Sep 14, 2022 at 11:13 AM Jeremy Dyer <jdy...@gmail.com> wrote: > > Hi, > As James mentioned in your example the "system" memory would be used by the > process/script in this case. > > I wanted to add that if you use the "ExecuteScript" processor that the NiFi > memory would be used. This is because NiFi uses Jython for processing > Python files. So where a native Cython process would typically create a > heap for a Python objects in a native process, Jython will instead use the > Java heap > > Jeremy Dyer > > On Wed, Sep 14, 2022 at 8:36 AM James Halfpenny <j...@source321.com> wrote: > > > Hi, > > The memory you allocate to NiFi will typically be the maximum heap size > > for the Java process. If you spawn a separate Python process then it will > > not be bound by the JVM heap size, so in your example it would come from > > the 8GB allocated for the “system”. > > > > I would recommend if you see high resource usage on the server again you > > take note of which processes are consuming the resources. > > > > Kind regards, > > Jim > > > > > On 14 Sep 2022, at 14:18, never more <vkruko...@gmail.com> wrote: > > > > > > Hello! > > > Can you help me with one question - can't find any information :( > > > > > > We have a nifi on my server (16 CPU, 32 RAM), in config our system adm. > > > set ram usage for nifi = 24 Gb, rest 8 Gb - for the system. > > > In Nifi we execute many Python scripts by using ExecuteStreamCommand > > (just > > > set path of script on server in properties). > > > So the question is - which resources python script uses - Nifi (24 gb) > > or > > > system (8 gb)? > > > > > > Maybe its a little stupid question😅, but I am afraid that improper use > > of > > > resources may lead to a service failure, > > > because we already have several cases of service failure when we > > noticed > > > high utilization of resources (RAM 90-95% and CPU 80-90%) > >
