[
https://issues.apache.org/jira/browse/NIFI-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247046#comment-16247046
]
Koji Kawamura commented on NIFI-2835:
-------------------------------------
Hi [~Eulicny],
To forgo or not, the decision is up to you. However, while I was implementing
ConsumeAzureEventHub, I happened to look at Azure Event Hub client source code
a lot. Then I felt if we had to do the similar thing ourselves in NiFi
codebase, it'd be challenging..
So, I'd like you to try ConsumeAzureEventHub processor if possible, which is
available as a PR for reviewing now. Any review comment would be appreciated.
https://github.com/apache/nifi/pull/2264
If you find that is helpful and no need to do anything at GetAzureEventHub,
then this JIRA can be closed by simply add 'SeeAlso' and probably
'DeprecationNotice' annotation to GetAzureEventHub so that users can notice the
limitation and another solution. How do you think?
> GetAzureEventHub processor should leverage partition offset to better handle
> restarts
> -------------------------------------------------------------------------------------
>
> Key: NIFI-2835
> URL: https://issues.apache.org/jira/browse/NIFI-2835
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Joseph Percivall
> Assignee: Eric Ulicny
>
> The GetAzureEventHub processor utilizes the Azure client that consists of
> receivers for each partition. The processor stores them in a map[1] that gets
> cleared every time the processor is stopped[2]. These receivers have
> partition offsets which keep track of which message it's currently on and
> which it should receive next. So currently, when the processor is
> stopped/restarted, any tracking of which message is next to be received is
> lost.
> If instead of clearing the map each time, we hold onto the receivers, or kept
> track of the partitionId/Offsets when stopping, (barring any relevant
> configuration changes) the processor would restart exactly where it left off
> with no loss of data.
> This would work very well with NIFI-2826.
> [1]https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L122
> [2]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L229
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
