Hi Stanislav, Thanks for the feedback. Do you see an issue with using `null` as the default group id (as addressed by Jason in his response)? This default group id would not support offset commits and consumers would use `auto.offset.reset` config when there is no current offset.
Thanks. --Vahid From: Stanislav Kozlovski <stanis...@confluent.io> To: dev@kafka.apache.org Date: 07/20/2018 11:09 AM Subject: Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer I agree with Jason's notion that > implicit use of the empty group.id to commit offsets is more likely to be causing users unexpected problems than actually providing a useful capability. I was initially confused that this is the behavior when investigating a new-ish JIRA issue < https://issues.apache.org/jira/browse/KAFKA-6758 > about the same topic. So, +1 to deprecating "" as a group.id The question after that becomes what the *default* value should be - should we: a) treat an unconfigured group.id consumer as a sort of intermittent consumer where you don't store offsets at all (thereby making the user explicitly sign up for them) b) have a default value which makes use of them? I sort of like the former. @Dhruvil, thinking about it at a high-level - yes. I can't think of a situation where it makes sense to name something an empty string as far as I'm aware - to me it seems like potential for confusion On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram <rajinisiva...@gmail.com> wrote: > +1 to deprecate use of "" as group.id since it is odd to have a resource > name that you cannot set ACLs for. Agree, we have to support older clients > though. > > Thanks, > > Rajini > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson <ja...@confluent.io> > wrote: > > > Hi Vahid, > > > > Sorry for getting to this so late. I think there are two things here: > > > > 1. The use of "" as a groupId has always been a dubious practice at best. > > We definitely ought to deprecate its use in the client. Perhaps in the > next > > major release, we can remove support completely. However, since older > > clients depend on it, we may have to continue letting the broker support > it > > to some extent. Perhaps we just need to bump the OffsetCommit request API > > and only accept the offset commit for older versions. You probably have > to > > do this anyway if you want to introduce the new error code since old > > clients will not expect it. > > > > 2. There should be a way for the consumer to indicate that it has no > group > > id and will not commit offsets. This is an explicit instruction that the > > consumer should not bother with coordinator lookup and such. We currently > > have some brittle logic in place to let users avoid the coordinator > lookup, > > but it is a bit error-prone. I was hoping that we could change the > default > > value of group.id to be null so that the user had to take an explicit > > action to opt into coordinator management (groups or offsets). However, > it > > is true that some users may be unknowingly depending on offset storage if > > they are using both the default group.id and the default > > enable.auto.commit. Perhaps one option is to disable enable.auto.commit > > automatically if no group.id is specified? I am not sure if there are > any > > drawbacks, but my feeling is that implicit use of the empty group.id to > > commit offsets is more likely to be causing users unexpected problems > than > > actually providing a useful capability. > > > > Thoughts? > > > > Thanks, > > Jason > > > > > > > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian < > > vahidhashem...@us.ibm.com> wrote: > > > > > Hi Viktor, > > > > > > Thanks for sharing your opinion. > > > So you're in favor of disallowing the empty ("") group id altogether > > (even > > > for fetching). > > > Given that ideally no one should be using the empty group id (at least > in > > > a production setting) I think the impact would be minimal in either > case. > > > > > > But as you said, let's hear what others think and I'd be happy to > modify > > > the KIP if needed. > > > > > > Regards. > > > --Vahid > > > > > > > > > > > > > > > From: Viktor Somogyi <viktorsomo...@gmail.com> > > > To: dev <dev@kafka.apache.org> > > > Date: 05/28/2018 05:18 AM > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id > > > behavior in KafkaConsumer > > > > > > > > > > > > Hi Vahid, > > > > > > (with the argument that using the default group id for offset commit > > > should not be the user's intention in practice). > > > > > > Yea, so in my opinion too this use case doesn't seem too practical. > Also > > I > > > think breaking the offset commit is not smaller from this perspective > > than > > > breaking fetch and offset fetch. If we suppose that someone uses the > > > default group id and we break the offset commit then that might be > harder > > > to detect than breaking the whole thing altogether. (If we think about > an > > > upgrade situation.) > > > So since we think it is not a practical use case, I think it would be > > > better to break altogether but ofc that's just my 2 cents :). Let's > > gather > > > other's input as well. > > > > > > Cheers, > > > Viktor > > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian < > > > vahidhashem...@us.ibm.com> wrote: > > > > > > > Hi Victor, > > > > > > > > Thanks for reviewing the KIP. > > > > > > > > Yes, to minimize the backward compatibility impact, there would be no > > > harm > > > > in letting a stand-alone consumer consume messages under a "" group > id > > > (as > > > > long as there is no offset commit). > > > > It would have to knowingly seek to an offset or rely on the > > > > auto.offset.reset config for the starting offset. > > > > This way the existing functionality would be preserved for the most > > part > > > > (with the argument that using the default group id for offset commit > > > > should not be the user's intention in practice). > > > > > > > > Does it seem reasonable? > > > > > > > > Thanks. > > > > --Vahid > > > > > > > > > > > > > > > > > > > > From: Viktor Somogyi <viktorsomo...@gmail.com> > > > > To: dev <dev@kafka.apache.org> > > > > Date: 05/25/2018 04:56 AM > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id > > > > behavior in KafkaConsumer > > > > > > > > > > > > > > > > Hi Vahid, > > > > > > > > When reading your KIP I coldn't fully understand why did you decide > at > > > > failing with "offset_commit" in case #2? Can't we fail with an empty > > > group > > > > id even in "fetch" or "fetch_offset"? What was the reason for > deciding > > > to > > > > fail at "offset_commit"? Was it because of upgrade compatibility > > > reasons? > > > > > > > > Thanks, > > > > Viktor > > > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu <yuzhih...@gmail.com> > wrote: > > > > > > > > > Looks good to me. > > > > > -------- Original message --------From: Vahid S Hashemian < > > > > > vahidhashem...@us.ibm.com> Date: 5/23/18 11:19 AM (GMT-08:00) > To: > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the > > > default > > > > > group id behavior in KafkaConsumer > > > > > Hi Ted, > > > > > > > > > > Thanks for reviewing the KIP. I updated the KIP and introduced an > > > error > > > > > code for the scenario described. > > > > > > > > > > --Vahid > > > > > > > > > > > > > > > > > > > > > > > > > From: Ted Yu <yuzhih...@gmail.com> > > > > > To: dev@kafka.apache.org > > > > > Date: 04/27/2018 04:31 PM > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id > > > > > behavior in KafkaConsumer > > > > > > > > > > > > > > > > > > > > bq. If they attempt an offset commit they will receive an error. > > > > > > > > > > Can you outline what specific error would be encountered ? > > > > > > > > > > Thanks > > > > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian < > > > > > vahidhashem...@us.ibm.com> wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I have drafted a proposal for improving the behavior of > > > KafkaConsumer > > > > > when > > > > > > using the default group id: > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > > > > > > > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer > > > > > > The proposal based on the issue and suggestion reported in > > > KAFKA-6774. > > > > > > > > > > > > Your feedback is welcome! > > > > > > > > > > > > Thanks. > > > > > > --Vahid > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Best, Stanislav
