Jason Gustafson created KAFKA-4435:
--------------------------------------
Summary: Improve storage overhead of group metadata
Key: KAFKA-4435
URL: https://issues.apache.org/jira/browse/KAFKA-4435
Project: Kafka
Issue Type: Improvement
Components: consumer
Reporter: Jason Gustafson
The GroupMetadataManager serializes the full subscriptions and assignments of
all consumer group members for each generation as a single message. This is a
problem for large consumer groups with a large number of topics since each
member's subscription/assignment is serialized separately. So if you have n
consumers each subscribing to the same m topics, then the serialized message
will contain m*n subscribed topics. At a certain size, you end up exceeding the
max message size.
Some ideas for getting around this have been 1) turning on compression and 2)
adding regex support to the protocol. Both of these help, but maybe we should
question whether the subscriptions/assignments need to be written at all. The
reason to include this information in the log is basically it prevent a
rebalance on coordinator failover. After failover, the new coordinator can
consume the log and determine the full state of every group. The consumers in
the group simply send heartbeats to the new coordinator, once it is found.
In fact, preventing the rebalance is not really the main issue: it's ensuring
that the last generation can commit its offsets. If nothing were written to the
log, then the group would be recreated after failover from scratch and existing
members would not be able to commit offsets (since their generation would no
longer be valid). But the subscription/assignment is opaque to the coordinator
and is not actually used when committing offsets. All it really need is the
generation and the list of memberIds.
Supposing then that we removed the subscriptions/assignments from the group,
but retained the generation/memberId information, one loose end is servicing
the DescribeGroup request. After failover, we would no longer have the
subscription/assignment information we need to answer that request. One option
would be to trigger a rebalance after failover in order to repopulate it. The
previous generation would still be able to commit offsets before rejoining the
group. Potentially we could even delay this rebalance until we actually receive
a DescribeGroups request.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
