Dominik Benz created NIFI-3531:
----------------------------------
Summary: modify session.recover behaviour in nifi-jms-processors
to cope with high-traffic JMS
Key: NIFI-3531
URL: https://issues.apache.org/jira/browse/NIFI-3531
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Dominik Benz
As described in this mailing list post
http://apache-nifi-developer-list.39713.n7.nabble.com/session-recover-behaviour-in-nifi-jms-processor-td14940.html
the current implementation of nifi-jms-processor, especially of JMSConsumer
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSConsumer.java
causes frequent re-delivery of JMS messages due to the following behaviour:
1) several threads perform the JMS session callback in parallel
2) each callback performs a session.recover
3) during high traffic, the situation arises that the ACKs from another thread
may not (yet) have arrived at the JMS server
4) this implies that the pointer of session.recover will reconsume the
not-yet-acked message from another thread
I understood Nifi prefers message duplication over message loss - however, in
our case the number of re-delivered messages is "piling up" over time, leading
to a growing number of un-acked messages in JMS and finally to storage problems
on the JMS server side.
It would be great to modify the nifi-jms-processor package to reliably cope
with higher-traffic JMS sources. Ideas from my side would be / include:
1) make synchronous / asynchronous delivery configurable
2) add configuration option for custom ACKing modes (i.e. non-standard JMS
modes like individual ACKing or NO_ACK)
3) add configuration option for JMS message selectors
>From other projects, I found the JMS communication options in this
>spark-jms-receiver package
https://github.com/tbfenet/spark-jms-receiver
very helpful (in which situations are synchronous / asynchronous approaches
reliable, how to buffer messages before acking them, how/when to ack, ...).
Here's also a java port of a subset:
https://github.com/bernhardschaefer/spark-jms-receiver-java
For any mentioned option I'm also happy to help!
Best & thanks for a great piece of software,
Dominik
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
