[
https://issues.apache.org/jira/browse/NIFI-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15960067#comment-15960067
]
Joseph Niemiec commented on NIFI-2835:
--------------------------------------
WE have also found that we receive duplicates when running in clustered nifi is
due to the partition model of eventhub allowing the below... Hope is that this
patch will also solve this issue of multiple Receivers on the same partition.
'There can be at most 5 concurrent readers on a partition per consumer group; "
which is unlike Kafka which gave us a more simple 1 reader per partition per
consumer group model..
https://github.com/Azure/azure-event-hubs-java/blob/dev/ConsumingEvents.md#ownership-failover-and-epochs
> GetAzureEventHub processor should leverage partition offset to better handle
> restarts
> -------------------------------------------------------------------------------------
>
> Key: NIFI-2835
> URL: https://issues.apache.org/jira/browse/NIFI-2835
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Joseph Percivall
> Assignee: Joseph Niemiec
>
> The GetAzureEventHub processor utilizes the Azure client that consists of
> receivers for each partition. The processor stores them in a map[1] that gets
> cleared every time the processor is stopped[2]. These receivers have
> partition offsets which keep track of which message it's currently on and
> which it should receive next. So currently, when the processor is
> stopped/restarted, any tracking of which message is next to be received is
> lost.
> If instead of clearing the map each time, we hold onto the receivers, or kept
> track of the partitionId/Offsets when stopping, (barring any relevant
> configuration changes) the processor would restart exactly where it left off
> with no loss of data.
> This would work very well with NIFI-2826.
> [1]https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L122
> [2]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L229
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
