Sorry, just noticed the typo. Instead of “limit the possibility of a message loads. . .” should be "limit the possibility of a message loss…"
Cheers Oleg > On Feb 24, 2017, at 9:28 AM, Oleg Zhurakousky <[email protected]> > wrote: > > Dominic > > Great to hear from you! > As you can see from the inline comment in the code, the recover is there for > a reason primarily to ensure or should I say limit the possibility of a > message loads in the event of a processor and/or NiFi crash. > As you may be aware, in NiFi we do prefer message duplication over message > loss. That said, I do see several possibilities for improvement especially > for the high traffic scenarios you are describing. One such improvement would > be to create a listening container version of ConsumeJMS which has far more > control over threading and session caching/sharing. > > Would you mind racing a JIRA issue - > https://issues.apache.org/jira/browse/NIFI/ describing everything you just > did and we’ll handle it. > > Cheers > Oleg > >> On Feb 24, 2017, at 9:08 AM, Dominik Benz <[email protected]> wrote: >> >> Hi, >> >> we're currently using Nifi to consume a relatively high-traffic JMS topic >> (40-60 messages per second). >> >> Worked well in principle - however, we then noticed that the the outbound >> rate (i.e. the number of messages we fetched) of the topic was consistently >> slightly higher than its inbound rate (i.e. the actual number of messages >> sent to the topic). This puzzled me, because (being the only subscriber to >> the topic) I would expect inbound and outbound traffic to be identical >> (given we can consume fast enough, which we can). >> >> Digging deeper, I found that in >> >> org.apache.nifi.jms.processors.JMSConsumer >> >> the method "consume" performs a session.recover: >> >> >> >> session.recover (as written in the comment) basically stops message delivery >> and re-starts from the last non-acked message. However, I think this leads >> to the following issue in high-traffic contexts: >> >> 1) several threads perform the JMS session callback in parallel >> 2) each callback performs a session.recover >> 3) during high traffic, the situation arises that the ACKs from another >> thread may not (yet) have arrived at the JMS server >> 4) this implies that the pointer of session.recover will reconsume the >> not-yet-acked message from another thread >> >> For verification, I performed so far the following steps: >> >> (a) manual implementation of a simplistic synchronous JMS topic consumer -> >> inbound/outbound identical as expected >> (b) patched nifi-jms-processors and commented out session.recover() - >> inbout/outbound identical as expected >> >> Any thoughts on this? My current impression is that session.recover in its >> current usage doesn't play well together with the multi-threaded JMS >> consumers. Or do I have any misconception? >> >> Thanks & best regards, >> Dominik >> >> >> >> -- >> View this message in context: >> http://apache-nifi-developer-list.39713.n7.nabble.com/session-recover-behaviour-in-nifi-jms-processor-tp14940.html >> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com. >> >
