GitHub user grantneale opened a pull request:
https://github.com/apache/kafka/pull/3158
KAFKA-5337: LagBasedAssignor partition assignment strategy
Existing partition assignment strategies (RangeAssignor and
RoundRobinAssignor) do not account for the current consumer group lag on each
partition. This can result in sub-optimal assignments when the distribution of
lags for a given topic and consumer group is skewed.
The LagBasedAssignor operates on a per-topic basis, and attempts to assign
partitions such that lag is distributed as evenly across a consumer group.
## Algorithm
For each topic, first obtain the lag on all partitions. Lag on a given
partition is the difference between the end offset and the last offset
committed by the consumer group. If no offsets have been committed for a
partition we determine the lag based on the code auto.offset.reset property. If
auto.offset.reset=latest, we assume a lag of 0. If auto.offset.reset=earliest
(or any other value) we assume lag equal to the total number of message
currently available in that partition.
Next, create a map storing the current total lag of all partitions assigned
to each member of the consumer group. Partitions are assigned in decreasing
order of lag, with each partition assigned to the consumer with least total
number of assigned partitions, breaking ties by assigning to the consumer with
the least total currently assigned lag.
Assigning partitions evenly across consumers (by partition count) ensures
that the assignment is reasonably balanced (by partition count) when all
partitions have a current lag of 0 or if the distribution of lags is heavily
skewed. It also gives the consumer group the best possible chance of remaining
balanced if the assignment is retained for a long period (assuming throughput
is consistent across members of the consumer group).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/grantneale/kafka
feature/kafka-5337-lag-based-assignor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3158.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3158
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
