[
https://issues.apache.org/jira/browse/KAFKA-8799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906627#comment-16906627
]
Vinoth Chandar commented on KAFKA-8799:
---------------------------------------
[~guozhang] filed this for the longer term fix.
> Support ability to pass global user data to consumers during Assignment
> -----------------------------------------------------------------------
>
> Key: KAFKA-8799
> URL: https://issues.apache.org/jira/browse/KAFKA-8799
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Vinoth Chandar
> Priority: Major
>
> This is a follow up from KAFKA-7149
> *Background :*
> Although we reduced the size of the AssignmentInfo object sent during each
> rebalance from leader to all followers in KAFKA-7149, we still repeat the
> same _partitionsByHost_ map for each host (all this when interactive queries
> are enabled) and thus still end up sending redundant bytes to the broker and
> also logging a large kafka message.
> With 100s of streams instances, this overhead can grow into tens of megabytes
> easily.
> *Proposal :*
> Extend the group assignment protocol to be able to support passing of an
> additional byte[], which can now contain the HostInfo ->
> partitions/partitionsByHost data just one time.
> {code}
> final class GroupAssignment {
> private final Map<String, Assignment> assignments;
> // bytes sent to each consumer from leader
> private final byte[] globalUserData
> ...
> }
> {code}
>
> This can generally be handy to any other application like Streams, that does
> some stateful processing or lightweight cluster management
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
