[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054557#comment-17054557
]
Peter Turcsanyi commented on NIFI-5184:
---
Retested on NiFi 1.11.3 and IBM MQ 9.0.0.0.
JMSConnectionFactoryProvider controller service works properly with IBM MQ
local bindings mode using the native libmqjbnd.so library (tested with multiple
CS instances).
JMSConnectionFactoryProvider settings:
||Property||Value||
|MQ ConnectionFactory Implementation|com.ibm.mq.jms.MQConnectionFactory|
|MQ Client Libraries path (i.e.
/usr/jms/lib)|/opt/mqm/java/lib/com.ibm.mq.allclient.jar,/opt/mqm/java/lib64/libmqjbnd.so|
|Broker URI| |
|SSL Context Service| |
|queueManager|myQMgr|
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An
> exception occurred in the Java(tm) MQI.
> at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source)
> at
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993536#comment-16993536
]
Peter Turcsanyi commented on NIFI-5184:
---
NIFI-6884 should solve this issue. It needs to be retested when NIFI-6884 is
ready.
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An
> exception occurred in the Java(tm) MQI.
> at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> com.ibm.msg.client.commonservices.j2se.NLSServices.createException(NLSServices.java:319)
> at
> com.ibm.msg.client.commonservices.nls.NLSServices.createException(NLSServices.java:226)
> at
>
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481694#comment-16481694
]
Sivaprasanna Sethuraman commented on NIFI-5184:
---
Just wanted to update that as part of NIFI-5073 (PR #2653), CLIENT_LIB_DIR_PATH
has been updated with dynamicallyModifiesClasspath(true)
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An
> exception occurred in the Java(tm) MQI.
> at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> com.ibm.msg.client.commonservices.j2se.NLSServices.createException(NLSServices.java:319)
> at
> com.ibm.msg.client.commonservices.nls.NLSServices.createException(NLSServices.java:226)
> at
>
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472367#comment-16472367
]
Bryan Bende commented on NIFI-5184:
---
Given the limitations of the JDK native library loading and the native
libraries themsevles, there is always going to be some trade-off... if we did
what you are suggesting then we would possible make the native libraries work
better, but would be sacrificing some of the isolation by making additional
native libs/jars visible at a higher level which makes them visible to other
components that don't need/want them.
Also, something to keep in mind is that not only would the native libs need to
be loaded at a higher shared level, but also the Java code that uses them...
One time i did a test where I tried to load the Hadoop native libs into the
system class loader right at the beginning of NiFi start up, but the Java code
was still loaded down in the NAR class loader, and that Java got unsatisfied
link errors because it also needed to be loaded into the system class loader
where the native libs were loaded, which would then mean Hadoop libs would be
on the classpath of every single NAR/component which then defeats the purpose
of NARs.
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
>
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472328#comment-16472328
]
Greg Senia commented on NIFI-5184:
--
[~joewitt] and [~bende] should there really be a shared classloader just like
in j2ee and with what they are doing with Spark. I'd even take a custom
property flag to load these jars at a higher shared classloader vs risking the
double loading..
[https://github.com/apache/spark/pull/5851]
https://issues.apache.org/jira/browse/SPARK-7819
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An
> exception occurred in the Java(tm) MQI.
> at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
>
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472080#comment-16472080
]
Greg Senia commented on NIFI-5184:
--
[https://www.oracle.com/search/results?Ntt=native%20library%20classloader=1=1=bugs=S3]
There are the rest of the native lib bugs/mostly feature requests to handle
multiple
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An
> exception occurred in the Java(tm) MQI.
> at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> com.ibm.msg.client.commonservices.j2se.NLSServices.createException(NLSServices.java:319)
> at
>
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472062#comment-16472062
]
Greg Senia commented on NIFI-5184:
--
[~joewitt] here is the original bug that folks chased way back when I was just
getting involved with WAS 3.5 on Solaris.
[https://bugs.java.com/view_bug.do?bug_id=4225434]
fred.oliver@East 2001-08-03 Will not fix. There is no API to force the
collection of any dead object to occur at any particular time. The ability of
the VM to force the unloading of a shared library is not something that can be
guaranteed on every platform. Anand's workaround (above) seems practical.
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An
> exception occurred in the Java(tm) MQI.
> at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source)
> at
>
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472051#comment-16472051
]
Joseph Witt commented on NIFI-5184:
---
[~gss2002] there are some client libs now written by wonderful people that if I
ever meet them I will hug. but the gist is they use some mechanism of prefixing
the loading of the native lib so it could have a unique name/be loaded multiple
times and use nicely across class loaders. In fairness though this feels like
something that should be fixed in the JVM so that native libs are bound to the
classloader which loaded them (but i'm pretty ignorant to details of what is
happening under the covers there so maybe that isnt feasible).
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An
> exception occurred in the Java(tm) MQI.
> at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source)
> at
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472044#comment-16472044
]
Greg Senia commented on NIFI-5184:
--
[~joewitt] and [~bende] that sounds like a valid solution. I know from my days
of doing WAS/Tomcat/JBoss with MQ and other nativelibs there was no way around
it other than having only one version of MQ on a node in a process at time. Or
in a vendor app that used a native lib only one version of the app/native lib
per jvm
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An
> exception occurred in the Java(tm) MQI.
> at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
>
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472016#comment-16472016
]
Bryan Bende commented on NIFI-5184:
---
Correct, that is what I was trying to say in my last sentence... basically you
should be able to have one JMS service or one HDFS processor that uses native
libs, but more than one is problematic because of what you said.
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An
> exception occurred in the Java(tm) MQI.
> at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> com.ibm.msg.client.commonservices.j2se.NLSServices.createException(NLSServices.java:319)
> at
>
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472008#comment-16472008
]
Joseph Witt commented on NIFI-5184:
---
specifically the hadoop/jms example you noted would still not work because the
client libs (hadoop/jms) in this case are not written to have favorable
classloader friendly behaviors (since native libs are JVM wide by name), right?
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An
> exception occurred in the Java(tm) MQI.
> at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> com.ibm.msg.client.commonservices.j2se.NLSServices.createException(NLSServices.java:319)
> at
>
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472006#comment-16472006
]
Bryan Bende commented on NIFI-5184:
---
In this case I think changing JmsConnectionFactoryProvider to remove all of the
class loading code that is inside the class, and then change the property for
client lib dir path to have dynamicallyModifiesClasspath(true) might help...
The difference is that by using the framework level capabilities it will only
modify the classpath when the value of the property changes, instead of on
every stop/start, and when it does change it will close the previous
ClassLoader and create a new one, so hopefully the native libs won't be seen as
already loaded.
The case that still won't work is if you try to have two or more instances of
JmsConnectionFactoryProvider and they all want to use client libs, this will be
the same as the Hadoop scenario and won't work.
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471914#comment-16471914
]
Greg Senia commented on NIFI-5184:
--
[~joewitt] seems like NIFI-3673 has a similar issue and the user calls it out
but never responds:
Byunghwa Yun added a comment - 13/Apr/17 03:54
Bryan Bende It's working. Thanks for your efforts.
But I got the another classloader problem. NiFi doesn't load the hadoop native
library.
I attach the log. Thank you.
2017-04-13 16:49:26,430 DEBUG [StandardProcessScheduler Thread-6]
org.apache.hadoop.util.NativeCodeLoader Trying to load the custom-built
native-hadoop library...
2017-04-13 16:49:26,430 DEBUG [StandardProcessScheduler Thread-6]
*org.apache.hadoop.util.NativeCodeLoader Failed to load native-hadoop with
error: java.lang.UnsatisfiedLinkError: Native Library
/home/hadoop/hadoop-2.6.0-cdh5.5.1/lib/native/libhadoop.so already loaded in
another classloader*
*2017-04-13 16:49:26,430 DEBUG [StandardProcessScheduler Thread-6]*
org.apache.hadoop.util.NativeCodeLoader
java.library.path=/home/hadoop/hdfs/lib/native
2017-04-13 16:49:26,430 WARN [StandardProcessScheduler Thread-6]
org.apache.hadoop.util.NativeCodeLoader Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
>
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471476#comment-16471476
]
Joseph Witt commented on NIFI-5184:
---
Yeah I was noticing that by coincidence earlier today. [~markap14] and
[~bbende] what do you think? Happen to know why that is in there?
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An
> exception occurred in the Java(tm) MQI.
> at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> com.ibm.msg.client.commonservices.j2se.NLSServices.createException(NLSServices.java:319)
> at
> com.ibm.msg.client.commonservices.nls.NLSServices.createException(NLSServices.java:226)
> at
>
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471473#comment-16471473
]
Greg Senia commented on NIFI-5184:
--
[~joewitt] thanks for the quick response! If there is anything I can do to help
here let me know I do see in the Utils code where load/reload attempts to keep
adding a classloader I think?
The Utils.addResourcesToClasspath to be called on enable method in
JMSConnectionFactoryProvider*.* So I assume that means its called every time?
Is it possible to somehow check if there is already a classloader with those
classes loaded. I do get that you may want to call it each time as there may be
diff jms providers running inside a jvm. I also thought about just having these
specific Nifi nodes where a middlware team puts local QMGRs and just making
sure the classloader is at the system level?
|public void enable(ConfigurationContext context) throws
InitializationException {|
| |try {|
| |if (!this.configured) {|
| |if (logger.isInfoEnabled()) {|
| |logger.info("Configuring " + this.getClass().getSimpleName() + " for '"|
| |+
context.getProperty(CONNECTION_FACTORY_IMPL).evaluateAttributeExpressions().getValue()
+ "' to be connected to '"|
| |+ BROKER_URI + "'");|
| |}|
| |// will load user provided libraries/resources on the classpath|
|
|*Utils.addResourcesToClasspath*(context.getProperty(CLIENT_LIB_DIR_PATH).evaluateAttributeExpressions().getValue());|
| | |
| |this.createConnectionFactoryInstance(context);|
| | |
| |this.setConnectionFactoryProperties(context);|
| |}|
| |this.configured = true;|
| |} catch (Exception e) {|
| |logger.error("Failed to configure " + this.getClass().getSimpleName(), e);|
| |this.configured = false;|
| |throw new IllegalStateException(e);|
| |}|
| |}|
| |
| */|
| |static void addResourcesToClasspath(String path) {|
| |if (logger.isDebugEnabled()) {|
| |logger.debug("Adding additional resources from '" + path + "' to the
classpath.");|
| |}|
| |if (path == null) {|
| |throw new IllegalArgumentException("'path' must not be null");|
| |}|
| |File libraryDir = new File(path);|
| |if (libraryDir.exists() && libraryDir.isDirectory()) {|
| |String[] cpResourceNames = libraryDir.list();|
| |URL[] urls = new URL[cpResourceNames.length];|
| |try {|
| |for (int i = 0; i < urls.length; i++) {|
| |urls[i] = new File(libraryDir, cpResourceNames[i]).toURI().toURL();|
| |if (logger.isDebugEnabled()) {|
| |logger.debug("Identifying additional resource to the classpath: " +
urls[i]);|
| |}|
| |}|
| |} catch (Exception e) {|
| |throw new IllegalStateException(|
| |"Failed to parse user libraries from '" + libraryDir.getAbsolutePath() +
"'", e);|
| |}|
| | |
| |URLClassLoader cl = new URLClassLoader(urls, Utils.class.getClassLoader());|
| |Thread.currentThread().setContextClassLoader(cl);|
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471446#comment-16471446
]
Joseph Witt commented on NIFI-5184:
---
It is a known issue that once you load a native lib it is going to stay there
and you cannot reload that same library (by name anyway) anywhere else across
the JVM. This JVM behavior then imposes a limitation on how far our
classloader isolation can go - native libs can be a problem.
However, since you point out this happens during enable/disable of the
controller service I'm wondering if we have instance classloading in this
chain. If we do we might be able to stop doing that (not sure) and then it
would be fine since it would only load it once (unless something else already
loaded it). If something else also loaded that native library then one option
would be to exclude whatever else is doing that if we can since it should only
need to be in one place.
NiFi's classloader model does ensure there are a range of important and shared
across all classloader classes. We of course try to minimize that to keep the
footprint of possible class friction low. Native libs though...argh...dont
afford much flexibility with the way they're handled as far as we can tell just
yet.
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
>
[jira] [Commented] (NIFI-5184) Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
[
https://issues.apache.org/jira/browse/NIFI-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471438#comment-16471438
]
Greg Senia commented on NIFI-5184:
--
Is it possible to do something similar as the Spark guys did with
sharedPrefixes basically classes that should be shared within the JVM vs having
seperate classloaders for each instance?
| Creates a [[HiveClient]] using a classloader that works according to the
following rules:|
| | * - Shared classes: Java, Scala, logging, and Spark classes are delegated
to `baseClassLoader`|
| | * allowing the results of calls to the [[HiveClient]] to be visible
externally.|
| | * - Hive classes: new instances are loaded from `execJars`. These classes
are not|
| | * accessible externally due to their custom loading.|
| | * - [[HiveClientImpl]]: a new copy is created for each instance of
`IsolatedClassLoader`.|
| | * This new instance is able to see a specific version of hive without using
reflection. Since|
| | * this is a unique instance, it is not visible externally other than as a
generic|
| | * [[HiveClient]], unless `isolationOn` is set to `false`.|
> Nifi JMS Controller Disable/Enable causes Classloader Issues with Native Libs
> -
>
> Key: NIFI-5184
> URL: https://issues.apache.org/jira/browse/NIFI-5184
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Greg Senia
>Priority: Major
>
> When attempting to use a Local IBM QMGR with MQ Bindings. IBM States that
> within the JDK only one instance of the native library can be loaded. I've
> worked around this issue by symlinking the IBM MQ Libs into nifi/lib/ which
> is not a good solution. Wondering if this is a known issue with Nifi and the
> NAR classloader functions or if this is something that can be fixed so that
> Nifi can correctly work IBM MQ and MQBindings.
>
> This only occurs after you disable and than reenable the controller:
>
> 2018-05-10 21:10:34,865 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS -
> JMSConsumer[destination:null; pub-sub:false;] failed to process session due
> to org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.: {}
> org.springframework.jms.UncategorizedJmsException: Uncategorized exception
> occured during JMS processing; nested exception is
> com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception
> occurred in the Java(tm) MQI.
> The Java(tm) MQI has thrown an exception describing the problem.
> See the linked exception for further information.; nested exception is
> com.ibm.mq.jmqi.local.LocalMQ$4: CC=2;RC=2495;AMQ8598: Failed to load the
> WebSphere MQ native JNI library: 'mqjbnd'.
> at
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
> at
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:497)
> at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
> at
> org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
> at
> org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
> at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
