Sergio, The reason these processors have this warning is because JMS requires that the contents of the message be fully loaded into memory. So for PublishJMS, each FlowFile is loaded into memory in order to push to the JMS broker. For ConsumeJMS, the message is received as an in-memory byte array before being written to the NiFi repository.
Each Thread/Concurrent Task, then, will hold the contents of one FlowFile in memory. So if you’re sending 1 KB FlowFiles with 10 Concurrent Tasks, that’s going to be 10 KB of content held in memory, roughly. That’s fine. But if you’ve got 500 PublishJMS processors, each with 10 Concurrent Tasks, and each trying to send/receive 1 MB messages, then you’re talking about 5,000 messages being operated on concurrently and each taking 1 MB of heap - for a total of 5 GB of your heap. Then you may be in trouble. Does that help? Thanks -Mark On Mar 5, 2021, at 8:50 AM, Sergio M. <[email protected]<mailto:[email protected]>> wrote: In our Nifi, we currently have some pipelines that connect to MQ queues. In these flows, we use the ConsumeJMS and PublishJMS processors. When we execute several pipelines at the same time, we see that the level of resource consumption increases. In the PublishJMS bibliography (https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-jms-processors-nar/1.11.4/org.apache.nifi.jms. processors.PublishJMS /), there is a note regarding its memory consumption: "An instance of this component can cause high usage of this system resource. Multiple instances or high concurrency settings may result a degradation of performance." In the future, we will have many more new pipelines and the consumption impact on the Nifi servers will be even greater. Do you know any solution or configuration to help us deal with this? Thanks!
