[
https://issues.apache.org/jira/browse/KAFKA-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Henry Cai updated KAFKA-3890:
-----------------------------
Description:
Currently the task assignment in TaskAssignor is not deterministic. During
cluster restart or rolling restart, even though the participating worker nodes
are the same, but the TaskAssignor is not able to maintain a deterministic
mapping, so about 20% partitions will be reassigned which would cause state
repopulation on cluster restart time.
When the participating worker nodes are not changed, we really just want to
keep the old task assignment.
was:
Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, or
the default values are directly used. We need to make them configurable for
advanced users. For example, some default values may not work perfectly for
some scenarios:
https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar
to "StreamsConfig", which defines all related rocksDB options configs, that can
be passed as key-value pairs to "StreamsConfig".
> Kafka Streams: task assignment is not maintained on cluster restart or
> rolling restart
> --------------------------------------------------------------------------------------
>
> Key: KAFKA-3890
> URL: https://issues.apache.org/jira/browse/KAFKA-3890
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Henry Cai
> Assignee: Henry Cai
> Labels: api, newbie
>
> Currently the task assignment in TaskAssignor is not deterministic. During
> cluster restart or rolling restart, even though the participating worker
> nodes are the same, but the TaskAssignor is not able to maintain a
> deterministic mapping, so about 20% partitions will be reassigned which would
> cause state repopulation on cluster restart time.
> When the participating worker nodes are not changed, we really just want to
> keep the old task assignment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
