[
https://issues.apache.org/jira/browse/KAFKA-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-8019:
-----------------------------------
Labels: kip (was: )
> Better Scaling Experience for KStream
> -------------------------------------
>
> Key: KAFKA-8019
> URL: https://issues.apache.org/jira/browse/KAFKA-8019
> Project: Kafka
> Issue Type: New Feature
> Components: streams
> Reporter: Boyang Chen
> Assignee: Boyang Chen
> Priority: Major
> Labels: kip
>
> In our day-to-day work, we found it really hard to scale up a stateful stream
> application when its state store is very heavy. The caveat is that when the
> newly spinned hosts take ownership of some active tasks, so that they need to
> use non-trivial amount of time to restore the state store from changelog
> topic. The reassigned tasks would be available for unpredicted long time,
> which is not favorable. Secondly the current global rebalance stops the
> entire application process, which in a rolling host swap scenario would
> suggest an infinite resource shuffling without actual progress.
> Following the community's [cooperative
> rebalancing|https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies]
> proposal, we need to build something similar for KStream to better handle
> the auto scaling experience.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
