[
https://issues.apache.org/jira/browse/KAFKA-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398118#comment-16398118
]
Matthias J. Sax commented on KAFKA-6645:
----------------------------------------
This should be supported already. On startup, Kafka Streams inspects it's local
state directory and adds available stores into rebalance metadata (ie,
prev-assigned standby tasks). This allows to reassign partitions accordingly to
avoid state migration.
Note, that upcoming 1.1 release contains some improvements to partition
assignment. This Jira might still be valid in order to improve the existing
strategy further. It would be great if you could try out 1.0 or better 1.1
release and check if Kafka Streams behaves as expected. If not, it would be
great to learn what is missing in detail so we can close those gaps.
Thanks a lot!
> Host Affinity to facilitate faster restarts of kafka streams applications
> -------------------------------------------------------------------------
>
> Key: KAFKA-6645
> URL: https://issues.apache.org/jira/browse/KAFKA-6645
> Project: Kafka
> Issue Type: New Feature
> Components: streams
> Reporter: Giridhar Addepalli
> Priority: Major
>
> Since Kafka Streams applications have lot of state in the stores in general,
> it would be good to remember the assignment of partitions to machines. So
> that when whole application is restarted for some reason, there is a way to
> use past assignment of partitions to machines and there won't be need to
> build up whole state by reading off of changelog kafka topic. This would
> result in faster start-up.
> Samza has support for Host Affinity
> ([https://samza.apache.org/learn/documentation/0.14/yarn/yarn-host-affinity.html])
> KIP-54
> ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy)]
> , handles cases where some members of consumer group goes down / comes up,
> and KIP-54 ensures there is minimal diff between assignments before and after
> rebalance.
> But to handle whole restart use case, we need to remember past assignment
> somewhere, and use it after restart.
> Please let us know if this is already solved problem / some cleaner way of
> achieving this objective
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
