[
https://issues.apache.org/jira/browse/KAFKA-8810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913396#comment-16913396
]
John Roesler commented on KAFKA-8810:
-------------------------------------
Hi all,
Offhand, it seems like a high-level registry would support many useful aspects
of governance in an enterprise context. For this specific purpose, it does seem
like a signature/digest of the topology would be sufficient, though.
Note that you could either have the members hash their topology and send the
hash in the JoinGroup request, in which case the leader could take some action
(like leaving the mis-matched member out of the assignment and returning an
error or something). Or, as suggested, you could have the leader return it's
digest of the topology in the JoinGroup response, and then members could
compare it to their versions and shut down or something if they don't match.
Not sure which side of the symmetry makes more sense.
Note that there are some other configurations that might be worth considering,
like if only one member is using EOS, or if a member has a different set of
serdes.
All of this notwithstanding, I'm wondering how this could possibly work with
rolling upgrades (i.e., the topology change is not an accident). Another
question is whether it's possible to also solve the related problem in which
you run an app, accidentally change the topology, and then restart the whole
cluster at once. Then, the leader wouldn't be able to save us, since it also
contains the mistake. Perhaps something like Patrik's idea is necessary for the
general case, I'm not sure.
WDYT?
> Add mechanism to detect topology mismatch between streams instances
> -------------------------------------------------------------------
>
> Key: KAFKA-8810
> URL: https://issues.apache.org/jira/browse/KAFKA-8810
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Vinoth Chandar
> Priority: Major
>
> Noticed this while reading through the StreamsPartitionAssignor related code.
> If an user accidentally deploys a different topology on one of the instances,
> there is no mechanism to detect this and refuse assignment/take action. Given
> Kafka Streams is designed as an embeddable library, I feel this is rather an
> important scenario to handle. For e.g, kafka streams is embedded into a web
> front end tier and operators deploy a hot fix for a site issue to a few
> instances that are leaking memory and that accidentally also deploys some
> topology changes with it.
> Please feel free to close the issue, if its a duplicate. (Could not find a
> ticket for this)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
