[
https://issues.apache.org/jira/browse/KAFKA-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16895357#comment-16895357
]
Levani Kokhreidze commented on KAFKA-8727:
------------------------------------------
I haven't looked at the code yet and have no idea how much work this gonna be
or if it's even possible to do. But as a Kafka Streams user who manages large
states, I clearly would love to see this feature.
> Control over standby tasks host assignment
> ------------------------------------------
>
> Key: KAFKA-8727
> URL: https://issues.apache.org/jira/browse/KAFKA-8727
> Project: Kafka
> Issue Type: New Feature
> Components: streams
> Reporter: Levani Kokhreidze
> Priority: Major
>
> *Motivation*
> As of now, Kafka Streams user has no control over to which host Kafka Streams
> application will create standby task. In production deployments (especially
> in Kubernetes) it's quite common to have multiple instances of the same Kafka
> Streams application deployed across more than one "cluster" in order to have
> high availability of the system.
> For example, if we have 6 Kafka Streams instances deployed across two
> clusters, we'll get 3 Kafka Streams instances per cluster. With the current
> implementation, Kafka Streams application may create "standby task" in the
> same cluster as the active task which, is not the most optimal solution,
> since, in case of cluster failure recovery time will be much bigger. This is
> especially problematic for Kafka Streams application that manages large state.
>
> *Possible Solution*
> It would be great if in the Kafka Streams configuration we could have a
> possibility to inject dynamic environment variables and use that environment
> variables to control over where standby task should be created.
> For example, suppose I have active task *1_1* with environment variable:
> *CLUSTER_ID: main01* then stnadby task for *1_1* should be created where
> *CLUSTER_ID* *!=* *main01*
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
