[
https://issues.apache.org/jira/browse/KAFKA-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Yu updated KAFKA-7132:
------------------------------
Comment: was deleted
(was: Well, when I was imagining this situation, I was thinking that the log's
maximum extent is at offset 70.)
> Consider adding faster form of rebalancing
> ------------------------------------------
>
> Key: KAFKA-7132
> URL: https://issues.apache.org/jira/browse/KAFKA-7132
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
> Reporter: Richard Yu
> Priority: Critical
> Labels: performance
>
> Currently, when a consumer falls out of a consumer group, it will restart
> processing from the last checkpointed offset. However, this design could
> result in a lag which some users could not afford to let happen. For example,
> lets say a consumer crashed at offset 100, with the last checkpointed offset
> being at 70. When it recovers at a later offset (say, 120), it will be behind
> by an offset range of 50 (120 - 70). This is because the consumer restarted
> at 70, forcing it to reprocess old data. To avoid this from happening, one
> option would be to allow the current consumer to start processing not from
> the last checkpointed offset (which is 70 in the example), but from 120 where
> it recovers. Meanwhile, a new KafkaConsumer will be instantiated and start
> reading from offset 70 in concurrency with the old process, and will be
> terminated once it reaches 120. In this manner, a considerable amount of lag
> can be avoided, particularly since the old consumer could proceed as if
> nothing had happened.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
