Hello Patricia, Could you provide more details about JVM properties that you tried configuring? Please also let us know the relevant versions of MiNiFi and NiFi you are using. If possible, providing the flow you are attempting to run would be helpful. Otherwise, if you could get into specifics about how the consume processor is configured would give a few more datapoints.
Additionally, I would like to request that you grab us a thread dump from MiNiFi when this issue is exhibited. This can be accomplished by running bin/minifi.sh dump <dump file name>. At its core, and beyond the initial configuration startup, MiNiFi is running primarily the same core libraries as NiFi and would expect similar JVM configurations to have similar success in operating. The inability to create new threads is a bit of a curious one that I have not seen especially when the same flow is not at issue in NiFi. Thanks! --aldrin On Fri, Sep 28, 2018 at 5:04 AM Patricia Quill <[email protected]> wrote: > Have deployed a ConsumeMQTT processor to a remote server using MiNiFi > (Java agent) and configured it to subscribe to a mosquitto broker. Problem > I am seeing is that it is causing the OS to run out of resources. > > 2018-09-27 09:06:37,060 WARN [Timer-Driven Process Thread-9] > o.a.n.controller.tasks.ConnectableTask Administratively Yielding > ConsumeMQTT[id=b3271a8d-cc15-3f10-0000-000000000000] due to uncaught > Exception: java.lang.OutOfMemoryError: unable to create new native thread > java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:717) > at > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) > at > java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603) > at > java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334) > at > java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) > at > java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:622) > at > org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.start(ClientComms.java:675) > at > org.eclipse.paho.client.mqttv3.internal.ClientComms.connect(ClientComms.java:280) > at > org.eclipse.paho.client.mqttv3.internal.ConnectActionListener.connect(ConnectActionListener.java:185) > at > org.eclipse.paho.client.mqttv3.MqttAsyncClient.connect(MqttAsyncClient.java:774) > at org.eclipse.paho.client.mqttv3.MqttClient.connect(MqttClient.java:333) > at > org.apache.nifi.processors.mqtt.common.AbstractMQTTProcessor.setAndConnectClient(AbstractMQTTProcessor.java:370) > at > org.apache.nifi.processors.mqtt.ConsumeMQTT.reconnect(ConsumeMQTT.java:347) > at > org.apache.nifi.processors.mqtt.ConsumeMQTT.onTrigger(ConsumeMQTT.java:256) > at > org.apache.nifi.processors.mqtt.common.AbstractMQTTProcessor.onTrigger(AbstractMQTTProcessor.java:355) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > > I've tried it on a number of different OS types - Ubuntu 14_04, Ubuntu > 16_04 and CentOS 7. I have tried tweaking the JVM properties but to no > avail. Am using NiFi 1.7.0. > The MiNiFi setup seems to be OK as I tried a test with the TailFile > processor and had no issues. The ConsumeMQTT processor works without any > issues if it's deployed on the NiFi server. > > Has anyone else come across this or have any suggestions? >
