Joseph Percivall created NIFI-2835:
--------------------------------------
Summary: GetAzureEventHub processor should leverage partition
offset to better handle restarts
Key: NIFI-2835
URL: https://issues.apache.org/jira/browse/NIFI-2835
Project: Apache NiFi
Issue Type: Improvement
Reporter: Joseph Percivall
The GetAzureEventHub processor utilizes the Azure client that consists of
receivers for each partition. The processor stores them in a map[1] that gets
cleared every time the processor is stopped[2]. These receivers have partition
offsets which keep track of which message it's currently on and which it should
receive next. So currently, when the processor is stopped/restarted, any
tracking of which message is next to be received is lost.
If instead of clearing the map each time, we hold onto the receivers, or kept
track of the partitionId/Offsets when stopping, (barring any relevant
configuration changes) the processor would restart exactly where it left off
with no loss of data.
This would work very well with NIFI-2826.
[1]https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L122
[2]
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/eventhub/GetAzureEventHub.java#L229
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
