[
https://issues.apache.org/jira/browse/NIFI-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854475#comment-15854475
]
Nicholas Carenza edited comment on NIFI-3437 at 2/6/17 6:13 PM:
----------------------------------------------------------------
[~joewitt] Apologies I am not very experienced with Kafka, or Nifi for that
matter. I was trying to find something in the ConsumeKafka processor that kept
track of state. I read up on the Kafka documentation a bit and I think I
understand how the processor works with Kafka to keep track of its offset. Let
me see if I have this right:
The processor commits its latest consumed offset to Kafka at least every
MAX_UNCOMMITTED_TIME and it's Kafka that keeps a record based on the consumer's
group id. So when I stop and start the processor, as long as it has the same
group id, Kafka will start sending messages starting from the last committed
offset.
https://kafka.apache.org/documentation/#distributionimpl
was (Author: ncarenza):
[~joewitt] Apologies I am not very experienced with Kafka, or Nifi for that
matter. I was trying to find something in the ConsumeKafka processor that kept
track of state. I read up on the Kafka documentation a bit and I think I
understand the processor works with Kafka to keep track of its offset. Let me
see if I have this right:
The processor commits its latest consumed offset to Kafka at least every
MAX_UNCOMMITTED_TIME and it's Kafka that keeps a record based on the consumer's
group id. So when I stop and start the processor, as long as it has the same
group id, Kafka will start sending messages starting from the last committed
offset.
https://kafka.apache.org/documentation/#distributionimpl
> KafkaConsumer pause/resume
> --------------------------
>
> Key: NIFI-3437
> URL: https://issues.apache.org/jira/browse/NIFI-3437
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Nicholas Carenza
> Priority: Minor
>
> Could the KafkaConsumer processor keep track of it's position so that if it
> is stopped and started again it can attempt to resume consuming from that
> position?
> If I have to restart nifi or rename the consumer processor I want to make
> sure I don't miss any messages.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
