[
https://issues.apache.org/jira/browse/NIFI-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245136#comment-16245136
]
Eric Ulicny commented on NIFI-2835:
-----------------------------------
Hello Koji,
I believe you're correct in that the EventHubProcessor Host is most likely the
way to go. Should I forgo working on this in favor of the ConsumeAzureEventHub
Processor? Optionally we could store a map of partition, enqueue time.
> GetAzureEventHub processor should leverage partition offset to better handle
> restarts
> -------------------------------------------------------------------------------------
>
> Key: NIFI-2835
> URL: https://issues.apache.org/jira/browse/NIFI-2835
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Joseph Percivall
> Assignee: Eric Ulicny
>
> The GetAzureEventHub processor utilizes the Azure client that consists of
> receivers for each partition. The processor stores them in a map[1] that gets
> cleared every time the processor is stopped[2]. These receivers have
> partition offsets which keep track of which message it's currently on and
> which it should receive next. So currently, when the processor is
> stopped/restarted, any tracking of which message is next to be received is
> lost.
> If instead of clearing the map each time, we hold onto the receivers, or kept
> track of the partitionId/Offsets when stopping, (barring any relevant
> configuration changes) the processor would restart exactly where it left off
> with no loss of data.
> This would work very well with NIFI-2826.
> [1]https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L122
> [2]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L229
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
