If you choose to manually assign topic partitions, then you won't be using the group protocol to dynamically manage partition assignments and thus don't have a need to poll or heartbeat at any interval. See "Manual Partition Assignment" in https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html .
On Thu, May 7, 2020 at 3:52 PM Boyuan Zhang <boyu...@apache.org> wrote: > Hi team, > > I'm building an application which uses Kafka Consumer APIs to read messages > from topics. I plan to manually assign TopicPartitions to my consumer and > seek a certain offset before starting to read. I'll also materialize the > last read offset and reuse it when creating the consumer later. > > Within my usage, I' curious whether I need to commit offset automatically > or manually. While going through the doc, it seems like committing offset > is only important to dynamic assignment. > > Another question around manual assignment is, is it still true that I need > to call poll() continuously to keep the consumer in the group described as > below? > > > It is also possible that the consumer could encounter a "livelock" > > situation where it is continuing to send heartbeats, but no progress is > > being made. To prevent the consumer from holding onto its partitions > > indefinitely in this case, we provide a liveness detection mechanism > using > > the max.poll.interval.ms setting. Basically if you don't call poll at > > least as frequently as the configured max interval, then the client will > > proactively leave the group so that another consumer can take over its > > partitions. When this happens, you may see an offset commit failure (as > > indicated by a CommitFailedException > > < > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/CommitFailedException.html> > thrown > > from a call to commitSync() > > < > https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#commitSync-- > >). > > This is a safety mechanism which guarantees that only active members of > the > > group are able to commit offsets. So to stay in the group, you must > > continue to call poll. > > What will happen to poll() with manual assignment if the consumer is > removed from the group? > > Thanks for your help! > > Boyuan >
