Dominic Great to hear from you! As you can see from the inline comment in the code, the recover is there for a reason primarily to ensure or should I say limit the possibility of a message loads in the event of a processor and/or NiFi crash. As you may be aware, in NiFi we do prefer message duplication over message loss. That said, I do see several possibilities for improvement especially for the high traffic scenarios you are describing. One such improvement would be to create a listening container version of ConsumeJMS which has far more control over threading and session caching/sharing.
Would you mind racing a JIRA issue - https://issues.apache.org/jira/browse/NIFI/ describing everything you just did and we’ll handle it. Cheers Oleg > On Feb 24, 2017, at 9:08 AM, Dominik Benz <[email protected]> wrote: > > Hi, > > we're currently using Nifi to consume a relatively high-traffic JMS topic > (40-60 messages per second). > > Worked well in principle - however, we then noticed that the the outbound > rate (i.e. the number of messages we fetched) of the topic was consistently > slightly higher than its inbound rate (i.e. the actual number of messages > sent to the topic). This puzzled me, because (being the only subscriber to > the topic) I would expect inbound and outbound traffic to be identical > (given we can consume fast enough, which we can). > > Digging deeper, I found that in > > org.apache.nifi.jms.processors.JMSConsumer > > the method "consume" performs a session.recover: > > > > session.recover (as written in the comment) basically stops message delivery > and re-starts from the last non-acked message. However, I think this leads > to the following issue in high-traffic contexts: > > 1) several threads perform the JMS session callback in parallel > 2) each callback performs a session.recover > 3) during high traffic, the situation arises that the ACKs from another > thread may not (yet) have arrived at the JMS server > 4) this implies that the pointer of session.recover will reconsume the > not-yet-acked message from another thread > > For verification, I performed so far the following steps: > > (a) manual implementation of a simplistic synchronous JMS topic consumer -> > inbound/outbound identical as expected > (b) patched nifi-jms-processors and commented out session.recover() - > inbout/outbound identical as expected > > Any thoughts on this? My current impression is that session.recover in its > current usage doesn't play well together with the multi-threaded JMS > consumers. Or do I have any misconception? > > Thanks & best regards, > Dominik > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/session-recover-behaviour-in-nifi-jms-processor-tp14940.html > Sent from the Apache NiFi Developer List mailing list archive at Nabble.com. >
