We're using Nifi a lot to interact with a JMS middleware (Tibco EMS); see also my previous post http://apache-nifi-developer-list.39713.n7.nabble.com/nifi-jms-processor-not-closing-connection-properly-in-quot-primary-node-only-quot-execution-mode-td15967.html
For the latter problem, the new possibilities of configuring shared (JMS 2.0) subscriptions introduced by https://issues.apache.org/jira/browse/NIFI-3983 seemed promising, especially for our clustered setup (3 Nifi instances, which could then share the load of JMS consumption equally). However, we are experiencing message loss in the following configuration of the ConsumeJMS Processor: PROPERTIES (...) destination type: TOPIC Acknowlegement mode: CLIENT_ACKNOWLEDGE durable subscription: FALSE shared subscription: TRUE subscription name: my-subscription SCHEDULING. (...) Strategy: timer-driven Concurrent Tasks: 1 Run Schedule: 0 sec Execution: Primary Node We're currently running Nifi 1.7.0. The message loss occurs for comparatively high-traffic topics. We'v established a controlled reproducible setup of sending 1000 messages, from which we are losing roughly 200-400. When increasing the concurrent tasks, we are loosing much less (even down to 0-9 messages), but still substantially. After checking the source code (especially https://github.com/apache/nifi/blob/rel/nifi-1.7.0/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSConsumer.java) and its integration of spring-jms, my first suspicion was that we hit a spring-jms problem similar to the one described here https://stackoverflow.com/questions/43083199/spring-jms-template-sync-receive-on-non-durable-subscriber-message-loss. We extended our test setup by a spring-jms consumer implementation similar to the one of Nifi; however, in this setup, we're consistently receiving all messages, which makes this bug as root cause less probable. Interestingly, in the following to situations, we do NOT experience message loss: (1) non-shared subscription (2) shared durable subscription Both are not really an option for our case, as (1) still has the problem when the primary node switches and (2) is currently not desired by our Tibco admins. My current impression is that the way how Nifi schedules / triggers the JMS consumption introduces little "breaks" in the shared non-durable JMS subscriptions; messages which occur during these breaks are consequently lost. Before going deeper into this direction by myself, it would be great to have an opinion if this direction is promising? Or any other thoughts / experiences on this topic? -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
