[
https://issues.apache.org/jira/browse/NIFI-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gardella Juan Pablo updated NIFI-7563:
--------------------------------------
Description:
Below an scenario to reproduce the non optimize usage of JMS resources. Suppose
it is required to publish 1 message to the destination {{D}} using
[PublishJMS|http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-jms-processors-nar/1.11.4/org.apache.nifi.jms.processors.PublishJMS/index.html].
The message is a flow file in the processor input queue.
It is important to know that internally the processor is using
[CachingConnectionFactory|https://github.com/spring-projects/spring-framework/blob/master/spring-jms/src/main/java/org/springframework/jms/connection/CachingConnectionFactory.java]
to reuse objects and a
[worker|https://github.com/apache/nifi/blob/a1b245e051245bb6c65e7b5ffc6ee982669b7ab7/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/AbstractJMSProcessor.java#L180]
to be able to use in thread safe manner. For JMS publishers, the default
configuration is to cache connections, sessions (only 1) and message producers.
*Preconditions*
# Flowfile has either {{jms_destination}} or {{jms_replyTo}} attribute
defined. Due to NIFI-7561, it should contain the word {{queue}} or {{topic}}.
Also notice {{jms_destination}} should be ignored, as suggested at NIFI-7564.
That will limit the scenario only when {{jms_replyTo}} attribute is defined.
# For simplicity, the processor is the first time it processes messages.
*Scenario*
# Processor picks the message. The
[worker|https://github.com/apache/nifi/blob/a1b245e051245bb6c65e7b5ffc6ee982669b7ab7/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/AbstractJMSProcessor.java#L180]
is created.
# Connection {{C1}} and session {{S1}} are created. The
[Message|https://docs.oracle.com/javaee/7/api/javax/jms/Message.html] {{M1_S1}}
is created and
[MessageProducer|https://docs.oracle.com/javaee/7/api/javax/jms/MessageProducer.html]
{{MP_S1}} created too. Required to deliver first message at
[JMSPublisher#publish|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSPublisher.java#L65].
# S1 and C1 are stored in {{CachingConnectionFactory}}. The caching connection
factory is created at
[AbstractJMSProcessor.java#L208|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/AbstractJMSProcessor.java#L208].
# An attempt to create a new connection and a new session are requested to the
connection factory to create destination defined in the header
{{jms_destination}} at
[JMSPublisher.java#L131|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSPublisher.java#L131].
Notice the connection {{C1}} is reused although *{{S1}} is not reused* (it is
required to check internal logic in CachingConnectionFactory to understand why
not). A new session {{S2}} is created and stored in the
{{CachingConnectionFactory}} as the new cached session.
# Message is published and {{S1}} and {{MP_S1}} are closed. As {{S1}} is not
in the cache, it is physically closed and {{MP_S1}}.
# At this point of time, the cached objects are {{C1}}, {{S2}}. *Ideally*, all
resources should be reused.
The scenario if it is applied to N consecutive messages create a lot of
sessions and message producers.
We found this issue by adding an
[Interceptor|https://activemq.apache.org/interceptors] to an [Apache ActiveMQ
v5.x|http://activemq.apache.org/components/classic/] broker to detect the
optimal usage of resources. For example, only one message producer per
connection. In below scenario we will be created N producers for the same
connection. Also in a Nifi flow that connects a
[ConsumeJMS|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-jms-processors-nar/1.11.4/org.apache.nifi.jms.processors.ConsumeJMS/]
with a PublishJMS. Notice {{ConsumeJMS}} populates by default
{{jms_destination}} flowfile attribute.
was:
Below an scenario to reproduce the non optimize usage of JMS resources. Suppose
it is required to publish 1 message to the destination {{D}} using
[PublishJMS|http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-jms-processors-nar/1.11.4/org.apache.nifi.jms.processors.PublishJMS/index.html].
The message is a flow file in the processor input queue.
It is important to know that internally the processor is using
[CachingConnectionFactory|https://github.com/spring-projects/spring-framework/blob/master/spring-jms/src/main/java/org/springframework/jms/connection/CachingConnectionFactory.java]
to reuse objects and a
[worker|https://github.com/apache/nifi/blob/a1b245e051245bb6c65e7b5ffc6ee982669b7ab7/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/AbstractJMSProcessor.java#L180]
to be able to use in thread safe manner. For JMS publishers, the default
configuration is to cache connections, sessions (only 1) and message producers.
*Preconditions*
# Flowfile has either {{jms_destination}} or {{jms_replyTo}} attribute
defined. Due to NIFI-7561, it should contain the word {{queue}} or {{topic}}.
Also notice {{jms_destination}} should be ignored, as suggested at NIFI-7564.
That will limit the scenario only when {{jms_replyTo}} attribute is defined.
# For simplicity, the processor is the first time it processes messages.
*Scenario*
# Processor picks the message. The
[worker|https://github.com/apache/nifi/blob/a1b245e051245bb6c65e7b5ffc6ee982669b7ab7/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/AbstractJMSProcessor.java#L180]
is created.
# Connection {{C1}} and session {{S1}} are created. The
[Message|https://docs.oracle.com/javaee/7/api/javax/jms/Message.html] {{M1_S1}}
is created and
[MessageProducer|https://docs.oracle.com/javaee/7/api/javax/jms/MessageProducer.html]
{{MP_S1}} created too. Required to deliver first message at
[JMSPublisher#publish|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSPublisher.java#L65].
# S1 and C1 are stored in {{CachingConnectionFactory}}. The caching connection
factory is created at
[AbstractJMSProcessor.java#L208|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/AbstractJMSProcessor.java#L208].
# An attempt to create a new connection and a new session are requested to the
connection factory to create destination defined in the header
{{jms_destination}} at
[JMSPublisher.java#L131|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSPublisher.java#L131].
Notice the connection {{C1}} is reused although *{{S1}} is not reused* (it is
required to check internal logic in CachingConnectionFactory to understand why
not). A new session {{S2}} is created and stored in the
{{CachingConnectionFactory}} as the new cached session.
# Message is published and {{S1}} and {{MP_S1}} are closed. As {{S1}} is not
in the cache, it is physically closed and {{MP_S1}}.
# At this point of time, the cached objects are {{C1}}, {{S2}}. *Ideally*, all
resources should be reused.
The scenario if it is applied to N consecutive messages create a lot of
sessions and message producers. We found this issue by adding an
[Interceptor|https://activemq.apache.org/interceptors] to an [Apache ActiveMQ
v5.x|http://activemq.apache.org/components/classic/] broker to detect the
optimal usage of resources. For example, only one message producer per
connection. In below scenario we will be created N producers for the same
connection.
> Optimize the usage of JMS sessions and message producers
> --------------------------------------------------------
>
> Key: NIFI-7563
> URL: https://issues.apache.org/jira/browse/NIFI-7563
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 1.6.0, 1.8.0, 1.7.1, 1.10.0, 1.9.2, 1.11.4
> Reporter: Gardella Juan Pablo
> Assignee: Gardella Juan Pablo
> Priority: Minor
>
> Below an scenario to reproduce the non optimize usage of JMS resources.
> Suppose it is required to publish 1 message to the destination {{D}} using
> [PublishJMS|http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-jms-processors-nar/1.11.4/org.apache.nifi.jms.processors.PublishJMS/index.html].
> The message is a flow file in the processor input queue.
> It is important to know that internally the processor is using
> [CachingConnectionFactory|https://github.com/spring-projects/spring-framework/blob/master/spring-jms/src/main/java/org/springframework/jms/connection/CachingConnectionFactory.java]
> to reuse objects and a
> [worker|https://github.com/apache/nifi/blob/a1b245e051245bb6c65e7b5ffc6ee982669b7ab7/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/AbstractJMSProcessor.java#L180]
> to be able to use in thread safe manner. For JMS publishers, the default
> configuration is to cache connections, sessions (only 1) and message
> producers.
> *Preconditions*
> # Flowfile has either {{jms_destination}} or {{jms_replyTo}} attribute
> defined. Due to NIFI-7561, it should contain the word {{queue}} or {{topic}}.
> Also notice {{jms_destination}} should be ignored, as suggested at NIFI-7564.
> That will limit the scenario only when {{jms_replyTo}} attribute is defined.
> # For simplicity, the processor is the first time it processes messages.
> *Scenario*
> # Processor picks the message. The
> [worker|https://github.com/apache/nifi/blob/a1b245e051245bb6c65e7b5ffc6ee982669b7ab7/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/AbstractJMSProcessor.java#L180]
> is created.
> # Connection {{C1}} and session {{S1}} are created. The
> [Message|https://docs.oracle.com/javaee/7/api/javax/jms/Message.html]
> {{M1_S1}} is created and
> [MessageProducer|https://docs.oracle.com/javaee/7/api/javax/jms/MessageProducer.html]
> {{MP_S1}} created too. Required to deliver first message at
> [JMSPublisher#publish|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSPublisher.java#L65].
> # S1 and C1 are stored in {{CachingConnectionFactory}}. The caching
> connection factory is created at
> [AbstractJMSProcessor.java#L208|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/AbstractJMSProcessor.java#L208].
> # An attempt to create a new connection and a new session are requested to
> the connection factory to create destination defined in the header
> {{jms_destination}} at
> [JMSPublisher.java#L131|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSPublisher.java#L131].
> Notice the connection {{C1}} is reused although *{{S1}} is not reused* (it
> is required to check internal logic in CachingConnectionFactory to understand
> why not). A new session {{S2}} is created and stored in the
> {{CachingConnectionFactory}} as the new cached session.
> # Message is published and {{S1}} and {{MP_S1}} are closed. As {{S1}} is not
> in the cache, it is physically closed and {{MP_S1}}.
> # At this point of time, the cached objects are {{C1}}, {{S2}}. *Ideally*,
> all resources should be reused.
> The scenario if it is applied to N consecutive messages create a lot of
> sessions and message producers.
> We found this issue by adding an
> [Interceptor|https://activemq.apache.org/interceptors] to an [Apache ActiveMQ
> v5.x|http://activemq.apache.org/components/classic/] broker to detect the
> optimal usage of resources. For example, only one message producer per
> connection. In below scenario we will be created N producers for the same
> connection. Also in a Nifi flow that connects a
> [ConsumeJMS|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-jms-processors-nar/1.11.4/org.apache.nifi.jms.processors.ConsumeJMS/]
> with a PublishJMS. Notice {{ConsumeJMS}} populates by default
> {{jms_destination}} flowfile attribute.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
