[ https://issues.apache.org/jira/browse/KAFKA-8799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906627#comment-16906627 ]
Vinoth Chandar commented on KAFKA-8799: --------------------------------------- [~guozhang] filed this for the longer term fix. > Support ability to pass global user data to consumers during Assignment > ----------------------------------------------------------------------- > > Key: KAFKA-8799 > URL: https://issues.apache.org/jira/browse/KAFKA-8799 > Project: Kafka > Issue Type: Improvement > Components: streams > Reporter: Vinoth Chandar > Priority: Major > > This is a follow up from KAFKA-7149 > *Background :* > Although we reduced the size of the AssignmentInfo object sent during each > rebalance from leader to all followers in KAFKA-7149, we still repeat the > same _partitionsByHost_ map for each host (all this when interactive queries > are enabled) and thus still end up sending redundant bytes to the broker and > also logging a large kafka message. > With 100s of streams instances, this overhead can grow into tens of megabytes > easily. > *Proposal :* > Extend the group assignment protocol to be able to support passing of an > additional byte[], which can now contain the HostInfo -> > partitions/partitionsByHost data just one time. > {code} > final class GroupAssignment { > private final Map<String, Assignment> assignments; > // bytes sent to each consumer from leader > private final byte[] globalUserData > ... > } > {code} > > This can generally be handy to any other application like Streams, that does > some stateful processing or lightweight cluster management > -- This message was sent by Atlassian JIRA (v7.6.14#76016)