[jira] [Commented] (KAFKA-8207) StickyPartitionAssignor for KStream
[ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901245#comment-16901245 ] neeraj commented on KAFKA-8207: --- Hi [~bchen225242] I have written you an email pasting content -- Initially, we got stuck in the critical bug https://issues.apache.org/jira/browse/KAFKA-8715 but then we took source code and changed the GroupMetadata.scala to use UUID instead of currentStateTimestamp (code snippet at the end of mail) as mentioned in the pull request. We rebuild the Kafka and replaced the kafka_2.12-2.3.0 jar at all broker lib folder We expected that when we restart our consumer cluster (20 machines)/or a single node we will be getting same partitions in the same node but that is not the case (consumer group give details). We need this because we plan to store a lot of aggregated data in rocks DB in our consumer cluster(20 machines)., different partitions to the different machine at restart means eventually each node will have all the data and it will take a lot of time to build the stores. > StickyPartitionAssignor for KStream > --- > > Key: KAFKA-8207 > URL: https://issues.apache.org/jira/browse/KAFKA-8207 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.0.0 >Reporter: neeraj >Priority: Major > > In KStreams I am not able to give a sticky partition assignor or my custom > partition assignor. > Overriding the property while building stream does not work > streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > CustomAssignor.class.getName()); > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (KAFKA-8207) StickyPartitionAssignor for KStream
[ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844168#comment-16844168 ] Boyang Chen commented on KAFKA-8207: Hey [~neeraj.bhatt] thank you for your interest! Good news is that we are planning to ship major features for static membership in 2.3, while unfortunately we could get everything in, and progress is tracked here: https://issues.apache.org/jira/browse/KAFKA-7018 Feel free to be our alpha user in case you are interested :) And grab me (boy...@confluent.io) if there is anything not behaving as expected. > StickyPartitionAssignor for KStream > --- > > Key: KAFKA-8207 > URL: https://issues.apache.org/jira/browse/KAFKA-8207 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.0.0 >Reporter: neeraj >Priority: Major > > In KStreams I am not able to give a sticky partition assignor or my custom > partition assignor. > Overriding the property while building stream does not work > streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > CustomAssignor.class.getName()); > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8207) StickyPartitionAssignor for KStream
[ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844062#comment-16844062 ] Matthias J. Sax commented on KAFKA-8207: I actually just talked to [~bchen225242] – it seems that static group membership will be included in 2.3 – the last missing piece was merged on Friday. The release plan wiki page just does not reflect this yet. > StickyPartitionAssignor for KStream > --- > > Key: KAFKA-8207 > URL: https://issues.apache.org/jira/browse/KAFKA-8207 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.0.0 >Reporter: neeraj >Priority: Major > > In KStreams I am not able to give a sticky partition assignor or my custom > partition assignor. > Overriding the property while building stream does not work > streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > CustomAssignor.class.getName()); > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8207) StickyPartitionAssignor for KStream
[ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843733#comment-16843733 ] Matthias J. Sax commented on KAFKA-8207: I think that's correct. [~bchen225242] worked hard to get it into 2.3, but I think it moved to 2.4. > StickyPartitionAssignor for KStream > --- > > Key: KAFKA-8207 > URL: https://issues.apache.org/jira/browse/KAFKA-8207 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.0.0 >Reporter: neeraj >Priority: Major > > In KStreams I am not able to give a sticky partition assignor or my custom > partition assignor. > Overriding the property while building stream does not work > streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > CustomAssignor.class.getName()); > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8207) StickyPartitionAssignor for KStream
[ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843677#comment-16843677 ] neeraj commented on KAFKA-8207: --- [~mjsax] KIP 345 seems is not a part of 2.3 release as given in [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820648] > StickyPartitionAssignor for KStream > --- > > Key: KAFKA-8207 > URL: https://issues.apache.org/jira/browse/KAFKA-8207 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.0.0 >Reporter: neeraj >Priority: Major > > In KStreams I am not able to give a sticky partition assignor or my custom > partition assignor. > Overriding the property while building stream does not work > streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > CustomAssignor.class.getName()); > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8207) StickyPartitionAssignor for KStream
[ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818742#comment-16818742 ] Matthias J. Sax commented on KAFKA-8207: There is no timeline yet. Details will be posted here: [https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan] (or follow the dev mailing list; the timeline should be announce soon). The high level plan is to release a new version every 4 month: [https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan] If you are interested to monitor progress on KIP-345 (or even help), you can follow the corresponding JIRAs: [https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances] – feel free to comment on those tickets if you want to help – PR reviews are always appreciated in any case! > StickyPartitionAssignor for KStream > --- > > Key: KAFKA-8207 > URL: https://issues.apache.org/jira/browse/KAFKA-8207 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.0.0 >Reporter: neeraj >Priority: Major > > In KStreams I am not able to give a sticky partition assignor or my custom > partition assignor. > Overriding the property while building stream does not work > streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > CustomAssignor.class.getName()); > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8207) StickyPartitionAssignor for KStream
[ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818670#comment-16818670 ] neeraj commented on KAFKA-8207: --- [~mjsax] When is 2.3 release planned? Please feel free to assign me tasks of KIP 345 if it helps in early release > StickyPartitionAssignor for KStream > --- > > Key: KAFKA-8207 > URL: https://issues.apache.org/jira/browse/KAFKA-8207 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.0.0 >Reporter: neeraj >Priority: Major > > In KStreams I am not able to give a sticky partition assignor or my custom > partition assignor. > Overriding the property while building stream does not work > streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > CustomAssignor.class.getName()); > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8207) StickyPartitionAssignor for KStream
[ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818471#comment-16818471 ] Matthias J. Sax commented on KAFKA-8207: I understand. However, this must be fixed differently. Not by allowing users to specify a custom partition assignor. Hence, I think we should close this ticket as "not a problem". > StickyPartitionAssignor for KStream > --- > > Key: KAFKA-8207 > URL: https://issues.apache.org/jira/browse/KAFKA-8207 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.0.0 >Reporter: neeraj >Priority: Major > > In KStreams I am not able to give a sticky partition assignor or my custom > partition assignor. > Overriding the property while building stream does not work > streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > CustomAssignor.class.getName()); > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8207) StickyPartitionAssignor for KStream
[ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817625#comment-16817625 ] neeraj commented on KAFKA-8207: --- [~mjsax] Thanks for explaining. Any idea when 2.3 is expected? also kindly assign me a bug related to this KIP if it helps in early release. > StickyPartitionAssignor for KStream > --- > > Key: KAFKA-8207 > URL: https://issues.apache.org/jira/browse/KAFKA-8207 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.0.0 >Reporter: neeraj >Priority: Major > > In KStreams I am not able to give a sticky partition assignor or my custom > partition assignor. > Overriding the property while building stream does not work > streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > CustomAssignor.class.getName()); > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8207) StickyPartitionAssignor for KStream
[
https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817133#comment-16817133
]
Matthias J. Sax commented on KAFKA-8207:
Thanks for following up!
{quote}StreamPartitionAssignor sticky policy is for the stickiness of Tasks to
partition. (StickyTaskAssignor)
{quote}
Well, StreamPartitionAssignor's stickiness is tailored to not move tasks (and
thus not move partitions), but assign tasks to the same node if possible. (What
you describe is also true, but this property is called "co-partitioning", not
"stickiness".)
{quote}When node 1 goes down (for rolling deployment or for some other reason)
rebalance kicks in and replica which is given by num.standby.replicas take over
so it is fault tolerant, but when node 1 comes back again rebalance again kicks
in and node 1 processes different partitions and updates the state store for
different partitions.
{quote}
This case should actually be covered by StreamPartitionAssignor and it should
migrate the original partitions/task back to node1.
Hence, we still don't want to allow users to overwrite
`StreamPartitionAssignor`. However, we know that `StreamPartitionAssignor` has
some issue for which the stickiness is not achieved. It seems you are hitting
this case. It's not easy to fix unfortunately.
What should help, is current work on KP-345:
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances]
It has a high change to be included in 2.3 release – however, it will also
require a broker upgrade.
> StickyPartitionAssignor for KStream
> ---
>
> Key: KAFKA-8207
> URL: https://issues.apache.org/jira/browse/KAFKA-8207
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 2.0.0
>Reporter: neeraj
>Priority: Major
>
> In KStreams I am not able to give a sticky partition assignor or my custom
> partition assignor.
> Overriding the property while building stream does not work
> streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
> CustomAssignor.class.getName());
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8207) StickyPartitionAssignor for KStream
[ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16816210#comment-16816210 ] neeraj commented on KAFKA-8207: --- StreamPartitionAssignor sticky policy is for the stickiness of Tasks to partition. (StickyTaskAssignor) I am looking for Stickiness of consumer to partition like in normal Kafka Consumer We have three consumers c1,c2, and c3 in the same consumer group using KStream (reading from a Kafka topic with multiple). Each consumer is in a separate machine. We are aggregating and saving it in the local state store at each node. When node 1 goes down (for rolling deployment or for some other reason) rebalance kicks in and replica which is given by num.standby.replicas take over so it is fault tolerant, but when node 1 comes back again rebalance again kicks in and node 1 processes different partitions and updates the state store for different partitions. So Ideally, we want that node 1 consumer c1 should work on the same partitions as it was working earlier so that we can save replay time from change log and disk space as because of frequent rebalancing local store has data for all the partitions problem similar to [https://tech.transferwise.com/achieving-high-availability-with-kafka-streams/] > StickyPartitionAssignor for KStream > --- > > Key: KAFKA-8207 > URL: https://issues.apache.org/jira/browse/KAFKA-8207 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.0.0 >Reporter: neeraj >Priority: Major > > In KStreams I am not able to give a sticky partition assignor or my custom > partition assignor. > Overriding the property while building stream does not work > streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > CustomAssignor.class.getName()); > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8207) StickyPartitionAssignor for KStream
[ https://issues.apache.org/jira/browse/KAFKA-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814976#comment-16814976 ] Matthias J. Sax commented on KAFKA-8207: That is by design. Kafka Streams' internal "StreamPartitionAssignor" also implements a "sticky" policy. I think we can close this ticket as invalid. What do you try to achieve and why do you want to change the partition assignor? > StickyPartitionAssignor for KStream > --- > > Key: KAFKA-8207 > URL: https://issues.apache.org/jira/browse/KAFKA-8207 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.0.0 >Reporter: neeraj >Priority: Major > > In KStreams I am not able to give a sticky partition assignor or my custom > partition assignor. > Overriding the property while building stream does not work > streams props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > CustomAssignor.class.getName()); > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
