Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol
If there is no additional feedback on this KIP I'll start a vote soon. Thanks. --Vahid From: "Vahid S Hashemian" To: dev@kafka.apache.org Date: 07/18/2018 04:47 PM Subject: Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol The KIP is updated to follow the suggestion of using consumer group generation. --Vahid From: "Vahid S Hashemian" To: dev@kafka.apache.org Date: 07/17/2018 02:32 PM Subject: Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol Hi Jason, Thanks for the feedback. That sounds good to me. I'll update the KIP accordingly and move the local generation option to "Rejected Alternatives". I'll send a note when the revised KIP is ready. --Vahid From: Jason Gustafson To: dev Date: 07/17/2018 02:22 PM Subject: Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol Hey Vahid, I'm with Mike that it seems simpler to just use the consumer group generation. Even if you can figure out how to reason about the local generation, it still seems confusing to have something called "generation" which is not the consumer group generation. There doesn't seem to be much downside to using the actual generation as far as I can tell and it is much easier to reason about. I was thinking we could expose it in PartitionAssignor.Assignment as an additional field. I think this wouldn't impact existing assignors, though it would allow for other stateful assignors to use the field in a similar way. It's a pity we didn't add a version to this schema, by the way. There's not much we can do if we can only add fields at the end of the schema, but I guess we're hoping this is the last change ;-) Thanks, Jason On Fri, Jul 13, 2018 at 10:07 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Mike, > > > > Thanks a lot for reviewing the KIP and sharing your feedback. > > I agree that such an issue could surface with option 1, but the existing > > PR (that currently implements this option) checks for such duplicate > > assignments and ignores one in favor of the other. So at the end there > > will be valid (non-duplicate) assignments to consumers, but they might > > deviate a bit from the ideal assignment. > > If rare scenarios like this are deemed troublesome and we want to avoid > > them, option 2 would probably be the way to go. In that case and in my > > opinion, option 2 would a better solution compared to introducing another > > field (e.g. timestamp). > > > > Regards. > > --Vahid > > > > > > > > From: Mike Freyberger > > To: "dev@kafka.apache.org" > > Date: 07/13/2018 08:42 PM > > Subject:Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data > > Protocol > > > > > > > > This is great! > > > > For the client side implementation, I think it’s still possible for there > > to be a duplication. I’ll try to walk through the example here. > > > > Let’s says there are 2 consumers, 1 topic with 2 partitions. > > > > After the initial rebalance, generation 0: > > Consumer A has partition 0 > > Consumer B has partition 1 > > > > Let’s say consumer B leaves the group (long debug, GC pause). This leads > > to another rebalance. This rebalance will be considered generation 1 and > > will result in: > > > > Generation 1, Consumer A owns partition 0,1 > > > > Now let’s say Consumer B is still out of the group and then Consumer A > > leaves as well. While Consumer A is out of the group, Consumer B rejoins > > the group. During this rebalance, the only previous state would be the > > initial generation 0 assignment. So this assignment would be considered > > generation 1 as well and would result in: > > > > Generation 1, Consumer B owns partition 0,1 > > > > When A rejoins the group, both consumers would claim ownership of both > > partitions and they would report the assignment was from generation 1. > > This gets us back into the same issue as before because the generation > > number cannot help at all. You could add a timestamp in addition to the > > generation marker, but that’d still be vulnerable to clock skew. > > > > Would hooking into the existing generation marker protect the assignor for > > this kind of situation? We need to make sure the selected implantation is > > protected against the kind of failure mentioned above. > > > > Also, I have been working on KIP-315, which is another Sticky Assignor, > > which also requires some kind of epoch/generation marker to be protected > > against zombies. So, I’d be in favor of a generic solutio
Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol
The KIP is updated to follow the suggestion of using consumer group generation. --Vahid From: "Vahid S Hashemian" To: dev@kafka.apache.org Date: 07/17/2018 02:32 PM Subject: Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol Hi Jason, Thanks for the feedback. That sounds good to me. I'll update the KIP accordingly and move the local generation option to "Rejected Alternatives". I'll send a note when the revised KIP is ready. --Vahid From: Jason Gustafson To: dev Date: 07/17/2018 02:22 PM Subject: Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol Hey Vahid, I'm with Mike that it seems simpler to just use the consumer group generation. Even if you can figure out how to reason about the local generation, it still seems confusing to have something called "generation" which is not the consumer group generation. There doesn't seem to be much downside to using the actual generation as far as I can tell and it is much easier to reason about. I was thinking we could expose it in PartitionAssignor.Assignment as an additional field. I think this wouldn't impact existing assignors, though it would allow for other stateful assignors to use the field in a similar way. It's a pity we didn't add a version to this schema, by the way. There's not much we can do if we can only add fields at the end of the schema, but I guess we're hoping this is the last change ;-) Thanks, Jason On Fri, Jul 13, 2018 at 10:07 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Mike, > > > > Thanks a lot for reviewing the KIP and sharing your feedback. > > I agree that such an issue could surface with option 1, but the existing > > PR (that currently implements this option) checks for such duplicate > > assignments and ignores one in favor of the other. So at the end there > > will be valid (non-duplicate) assignments to consumers, but they might > > deviate a bit from the ideal assignment. > > If rare scenarios like this are deemed troublesome and we want to avoid > > them, option 2 would probably be the way to go. In that case and in my > > opinion, option 2 would a better solution compared to introducing another > > field (e.g. timestamp). > > > > Regards. > > --Vahid > > > > > > > > From: Mike Freyberger > > To: "dev@kafka.apache.org" > > Date: 07/13/2018 08:42 PM > > Subject:Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data > > Protocol > > > > > > > > This is great! > > > > For the client side implementation, I think it’s still possible for there > > to be a duplication. I’ll try to walk through the example here. > > > > Let’s says there are 2 consumers, 1 topic with 2 partitions. > > > > After the initial rebalance, generation 0: > > Consumer A has partition 0 > > Consumer B has partition 1 > > > > Let’s say consumer B leaves the group (long debug, GC pause). This leads > > to another rebalance. This rebalance will be considered generation 1 and > > will result in: > > > > Generation 1, Consumer A owns partition 0,1 > > > > Now let’s say Consumer B is still out of the group and then Consumer A > > leaves as well. While Consumer A is out of the group, Consumer B rejoins > > the group. During this rebalance, the only previous state would be the > > initial generation 0 assignment. So this assignment would be considered > > generation 1 as well and would result in: > > > > Generation 1, Consumer B owns partition 0,1 > > > > When A rejoins the group, both consumers would claim ownership of both > > partitions and they would report the assignment was from generation 1. > > This gets us back into the same issue as before because the generation > > number cannot help at all. You could add a timestamp in addition to the > > generation marker, but that’d still be vulnerable to clock skew. > > > > Would hooking into the existing generation marker protect the assignor for > > this kind of situation? We need to make sure the selected implantation is > > protected against the kind of failure mentioned above. > > > > Also, I have been working on KIP-315, which is another Sticky Assignor, > > which also requires some kind of epoch/generation marker to be protected > > against zombies. So, I’d be in favor of a generic solution here that other > > assignors can leverage. > > > > Best, > > > > Mike Freyberger > > > > > On Jul 13, 2018, at 6:15 PM, Vahid S Hashemian > > wrote: > > > > > > Hi all, > > > > > &
Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol
Hi Jason, Thanks for the feedback. That sounds good to me. I'll update the KIP accordingly and move the local generation option to "Rejected Alternatives". I'll send a note when the revised KIP is ready. --Vahid From: Jason Gustafson To: dev Date: 07/17/2018 02:22 PM Subject: Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol Hey Vahid, I'm with Mike that it seems simpler to just use the consumer group generation. Even if you can figure out how to reason about the local generation, it still seems confusing to have something called "generation" which is not the consumer group generation. There doesn't seem to be much downside to using the actual generation as far as I can tell and it is much easier to reason about. I was thinking we could expose it in PartitionAssignor.Assignment as an additional field. I think this wouldn't impact existing assignors, though it would allow for other stateful assignors to use the field in a similar way. It's a pity we didn't add a version to this schema, by the way. There's not much we can do if we can only add fields at the end of the schema, but I guess we're hoping this is the last change ;-) Thanks, Jason On Fri, Jul 13, 2018 at 10:07 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Mike, > > > > Thanks a lot for reviewing the KIP and sharing your feedback. > > I agree that such an issue could surface with option 1, but the existing > > PR (that currently implements this option) checks for such duplicate > > assignments and ignores one in favor of the other. So at the end there > > will be valid (non-duplicate) assignments to consumers, but they might > > deviate a bit from the ideal assignment. > > If rare scenarios like this are deemed troublesome and we want to avoid > > them, option 2 would probably be the way to go. In that case and in my > > opinion, option 2 would a better solution compared to introducing another > > field (e.g. timestamp). > > > > Regards. > > --Vahid > > > > > > > > From: Mike Freyberger > > To: "dev@kafka.apache.org" > > Date: 07/13/2018 08:42 PM > > Subject:Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data > > Protocol > > > > > > > > This is great! > > > > For the client side implementation, I think it’s still possible for there > > to be a duplication. I’ll try to walk through the example here. > > > > Let’s says there are 2 consumers, 1 topic with 2 partitions. > > > > After the initial rebalance, generation 0: > > Consumer A has partition 0 > > Consumer B has partition 1 > > > > Let’s say consumer B leaves the group (long debug, GC pause). This leads > > to another rebalance. This rebalance will be considered generation 1 and > > will result in: > > > > Generation 1, Consumer A owns partition 0,1 > > > > Now let’s say Consumer B is still out of the group and then Consumer A > > leaves as well. While Consumer A is out of the group, Consumer B rejoins > > the group. During this rebalance, the only previous state would be the > > initial generation 0 assignment. So this assignment would be considered > > generation 1 as well and would result in: > > > > Generation 1, Consumer B owns partition 0,1 > > > > When A rejoins the group, both consumers would claim ownership of both > > partitions and they would report the assignment was from generation 1. > > This gets us back into the same issue as before because the generation > > number cannot help at all. You could add a timestamp in addition to the > > generation marker, but that’d still be vulnerable to clock skew. > > > > Would hooking into the existing generation marker protect the assignor for > > this kind of situation? We need to make sure the selected implantation is > > protected against the kind of failure mentioned above. > > > > Also, I have been working on KIP-315, which is another Sticky Assignor, > > which also requires some kind of epoch/generation marker to be protected > > against zombies. So, I’d be in favor of a generic solution here that other > > assignors can leverage. > > > > Best, > > > > Mike Freyberger > > > > > On Jul 13, 2018, at 6:15 PM, Vahid S Hashemian > > wrote: > > > > > > Hi all, > > > > > > I create a short KIP to address an issue in Sticky Assignor assignment > > > logic: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol > > > > > Please take a look and share your feedback / comments. > > > > > > In particular, there is a Generation Marker section ( > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol#KIP- > 341:UpdateStickyAssignor'sUserDataProtocol-GenerationMarker > > > > > ) that provides two methods for implementing the improvement to the > > > protocol. I'd like to know which option is more popular. > > > > > > Thanks! > > > --Vahid > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol
Hey Vahid, I'm with Mike that it seems simpler to just use the consumer group generation. Even if you can figure out how to reason about the local generation, it still seems confusing to have something called "generation" which is not the consumer group generation. There doesn't seem to be much downside to using the actual generation as far as I can tell and it is much easier to reason about. I was thinking we could expose it in PartitionAssignor.Assignment as an additional field. I think this wouldn't impact existing assignors, though it would allow for other stateful assignors to use the field in a similar way. It's a pity we didn't add a version to this schema, by the way. There's not much we can do if we can only add fields at the end of the schema, but I guess we're hoping this is the last change ;-) Thanks, Jason On Fri, Jul 13, 2018 at 10:07 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Mike, > > > > Thanks a lot for reviewing the KIP and sharing your feedback. > > I agree that such an issue could surface with option 1, but the existing > > PR (that currently implements this option) checks for such duplicate > > assignments and ignores one in favor of the other. So at the end there > > will be valid (non-duplicate) assignments to consumers, but they might > > deviate a bit from the ideal assignment. > > If rare scenarios like this are deemed troublesome and we want to avoid > > them, option 2 would probably be the way to go. In that case and in my > > opinion, option 2 would a better solution compared to introducing another > > field (e.g. timestamp). > > > > Regards. > > --Vahid > > > > > > > > From: Mike Freyberger > > To: "dev@kafka.apache.org" > > Date: 07/13/2018 08:42 PM > > Subject:Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data > > Protocol > > > > > > > > This is great! > > > > For the client side implementation, I think it’s still possible for there > > to be a duplication. I’ll try to walk through the example here. > > > > Let’s says there are 2 consumers, 1 topic with 2 partitions. > > > > After the initial rebalance, generation 0: > > Consumer A has partition 0 > > Consumer B has partition 1 > > > > Let’s say consumer B leaves the group (long debug, GC pause). This leads > > to another rebalance. This rebalance will be considered generation 1 and > > will result in: > > > > Generation 1, Consumer A owns partition 0,1 > > > > Now let’s say Consumer B is still out of the group and then Consumer A > > leaves as well. While Consumer A is out of the group, Consumer B rejoins > > the group. During this rebalance, the only previous state would be the > > initial generation 0 assignment. So this assignment would be considered > > generation 1 as well and would result in: > > > > Generation 1, Consumer B owns partition 0,1 > > > > When A rejoins the group, both consumers would claim ownership of both > > partitions and they would report the assignment was from generation 1. > > This gets us back into the same issue as before because the generation > > number cannot help at all. You could add a timestamp in addition to the > > generation marker, but that’d still be vulnerable to clock skew. > > > > Would hooking into the existing generation marker protect the assignor for > > this kind of situation? We need to make sure the selected implantation is > > protected against the kind of failure mentioned above. > > > > Also, I have been working on KIP-315, which is another Sticky Assignor, > > which also requires some kind of epoch/generation marker to be protected > > against zombies. So, I’d be in favor of a generic solution here that other > > assignors can leverage. > > > > Best, > > > > Mike Freyberger > > > > > On Jul 13, 2018, at 6:15 PM, Vahid S Hashemian > > wrote: > > > > > > Hi all, > > > > > > I create a short KIP to address an issue in Sticky Assignor assignment > > > logic: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol > > > > > Please take a look and share your feedback / comments. > > > > > > In particular, there is a Generation Marker section ( > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol#KIP- > 341:UpdateStickyAssignor'sUserDataProtocol-GenerationMarker > > > > > ) that provides two methods for implementing the improvement to the > > > protocol. I'd like to know which option is more popular. > > > > > > Thanks! > > > --Vahid > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol
Hi Mike, Thanks a lot for reviewing the KIP and sharing your feedback. I agree that such an issue could surface with option 1, but the existing PR (that currently implements this option) checks for such duplicate assignments and ignores one in favor of the other. So at the end there will be valid (non-duplicate) assignments to consumers, but they might deviate a bit from the ideal assignment. If rare scenarios like this are deemed troublesome and we want to avoid them, option 2 would probably be the way to go. In that case and in my opinion, option 2 would a better solution compared to introducing another field (e.g. timestamp). Regards. --Vahid From: Mike Freyberger To: "dev@kafka.apache.org" Date: 07/13/2018 08:42 PM Subject: Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol This is great! For the client side implementation, I think it’s still possible for there to be a duplication. I’ll try to walk through the example here. Let’s says there are 2 consumers, 1 topic with 2 partitions. After the initial rebalance, generation 0: Consumer A has partition 0 Consumer B has partition 1 Let’s say consumer B leaves the group (long debug, GC pause). This leads to another rebalance. This rebalance will be considered generation 1 and will result in: Generation 1, Consumer A owns partition 0,1 Now let’s say Consumer B is still out of the group and then Consumer A leaves as well. While Consumer A is out of the group, Consumer B rejoins the group. During this rebalance, the only previous state would be the initial generation 0 assignment. So this assignment would be considered generation 1 as well and would result in: Generation 1, Consumer B owns partition 0,1 When A rejoins the group, both consumers would claim ownership of both partitions and they would report the assignment was from generation 1. This gets us back into the same issue as before because the generation number cannot help at all. You could add a timestamp in addition to the generation marker, but that’d still be vulnerable to clock skew. Would hooking into the existing generation marker protect the assignor for this kind of situation? We need to make sure the selected implantation is protected against the kind of failure mentioned above. Also, I have been working on KIP-315, which is another Sticky Assignor, which also requires some kind of epoch/generation marker to be protected against zombies. So, I’d be in favor of a generic solution here that other assignors can leverage. Best, Mike Freyberger > On Jul 13, 2018, at 6:15 PM, Vahid S Hashemian wrote: > > Hi all, > > I create a short KIP to address an issue in Sticky Assignor assignment > logic: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol > Please take a look and share your feedback / comments. > > In particular, there is a Generation Marker section ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP-341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol#KIP-341:UpdateStickyAssignor'sUserDataProtocol-GenerationMarker > ) that provides two methods for implementing the improvement to the > protocol. I'd like to know which option is more popular. > > Thanks! > --Vahid > >
Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol
This is great! For the client side implementation, I think it’s still possible for there to be a duplication. I’ll try to walk through the example here. Let’s says there are 2 consumers, 1 topic with 2 partitions. After the initial rebalance, generation 0: Consumer A has partition 0 Consumer B has partition 1 Let’s say consumer B leaves the group (long debug, GC pause). This leads to another rebalance. This rebalance will be considered generation 1 and will result in: Generation 1, Consumer A owns partition 0,1 Now let’s say Consumer B is still out of the group and then Consumer A leaves as well. While Consumer A is out of the group, Consumer B rejoins the group. During this rebalance, the only previous state would be the initial generation 0 assignment. So this assignment would be considered generation 1 as well and would result in: Generation 1, Consumer B owns partition 0,1 When A rejoins the group, both consumers would claim ownership of both partitions and they would report the assignment was from generation 1. This gets us back into the same issue as before because the generation number cannot help at all. You could add a timestamp in addition to the generation marker, but that’d still be vulnerable to clock skew. Would hooking into the existing generation marker protect the assignor for this kind of situation? We need to make sure the selected implantation is protected against the kind of failure mentioned above. Also, I have been working on KIP-315, which is another Sticky Assignor, which also requires some kind of epoch/generation marker to be protected against zombies. So, I’d be in favor of a generic solution here that other assignors can leverage. Best, Mike Freyberger > On Jul 13, 2018, at 6:15 PM, Vahid S Hashemian > wrote: > > Hi all, > > I create a short KIP to address an issue in Sticky Assignor assignment > logic: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol > Please take a look and share your feedback / comments. > > In particular, there is a Generation Marker section ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP-341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol#KIP-341:UpdateStickyAssignor'sUserDataProtocol-GenerationMarker > ) that provides two methods for implementing the improvement to the > protocol. I'd like to know which option is more popular. > > Thanks! > --Vahid > >
[DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol
Hi all, I create a short KIP to address an issue in Sticky Assignor assignment logic: https://cwiki.apache.org/confluence/display/KAFKA/KIP-341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol Please take a look and share your feedback / comments. In particular, there is a Generation Marker section ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol#KIP-341:UpdateStickyAssignor'sUserDataProtocol-GenerationMarker ) that provides two methods for implementing the improvement to the protocol. I'd like to know which option is more popular. Thanks! --Vahid
