Hey Justine, Thanks for that. In a way you are right, that setting the new config to -1 is equivalent to setting it equal to transactional.id.expiration.ms . The only reason I mentioned the -1 case is it gives the users the ability to disable this feature (which mayn't even be desirable based on what Jason described above) and still be able to re-configure transactional.id.expiration.ms. Otherwise, everytime users change transactional.id.expiration.ms, they will need to also remember and change this new config. Again I don't know how frequently this config is changed or should users even disable this feature, but that was the reasoning I had. If you think it isn't necessary, we can remove it.
On Sat, Jul 30, 2022 at 3:08 AM Justine Olshan <jols...@confluent.io.invalid> wrote: > Hi all, > > I've updated the KIP to include the new default of 1 day and information > about -1 in the description of the config. > I wonder though if including -1 makes sense now that it is not the default > value. Is there a benefit for manually setting -1 vs manually setting the > value that transactional.id.expiration.ms has? > > Let me know your thoughts. > Thanks, > Justine > > On Fri, Jul 29, 2022 at 10:54 AM Ismael Juma <ism...@juma.me.uk> wrote: > > > +1 for having 1 day as the default and for including this change in the > > release notes. > > > > Ismael > > > > On Wed, Jul 27, 2022 at 9:16 AM Jason Gustafson > <ja...@confluent.io.invalid > > > > > wrote: > > > > > I don't think a major release is a requirement for a default change for > > > what it's worth. I do appreciate that there is a preference for not > > rocking > > > the boat though. For a little bit of background here, the problem we > > > have encountered in production since the idempotent producer became the > > > default is OOM errors due to huge numbers of producerIds that had to be > > > retained in the cache for 7 days. It is hard to prevent use cases from > > > emerging where producers are used and discarded rapidly. We will be > > using a > > > lower value for sure, but it would also be nice to reduce the > likelihood > > > for the community to see this problem. The benefit of the caching > > > diminishes quickly over time since it is primarily meant to handle > > producer > > > retry windows. I do not think there is much difference between 1 days > > and 7 > > > days from an application perspective, but it is a huge difference for > the > > > broker's memory usage. > > > > > > Best, > > > Jason > > > > > > On Wed, Jul 27, 2022 at 2:57 AM Sagar <sagarmeansoc...@gmail.com> > wrote: > > > > > > > Thanks Justine for the KIP. I think it might be better to document > the > > > > correlation between the new config and delivery.timeout.ms in the > > Public > > > > Interfaces Description. > > > > > > > > Also, I agree with Luke that for now setting a default to -1 should > be > > > > good. We can look to switch to 1 day with major release. > > > > > > > > Thanks! > > > > Sagar. > > > > > > > > On Wed, Jul 27, 2022 at 9:05 AM Luke Chen <show...@gmail.com> wrote: > > > > > > > > > Hi Justine, > > > > > > > > > > Thanks for the KIP. > > > > > I agree with you that we should try our best to keep backward > > > > > compatibility, although our intention is to have lower producer id > > > > > expiration timeout. > > > > > So, I think we should keep default to -1 IMO. > > > > > Maybe we change the default to 1 day in next major release (4.0)? > > > > > > > > > > Thank you. > > > > > Luke > > > > > > > > > > On Wed, Jul 27, 2022 at 7:13 AM Justine Olshan > > > > > <jols...@confluent.io.invalid> > > > > > wrote: > > > > > > > > > > > Thanks for taking a look Jason! > > > > > > > > > > > > I wondered if we wanted to have a smaller default but wasn't sure > > > about > > > > > the > > > > > > compatibility story -- especially since there is the chance for > > > > producer > > > > > > IDs to expire silently. > > > > > > I do think that 1 day is fairly reasonable. If I don't hear any > > > > > conflicting > > > > > > opinions I can go ahead and update the default. > > > > > > > > > > > > Justine > > > > > > > > > > > > On Tue, Jul 26, 2022 at 12:23 PM Jason Gustafson > > > > > > <ja...@confluent.io.invalid> > > > > > > wrote: > > > > > > > > > > > > > Hi Justine, > > > > > > > > > > > > > > Thanks for the KIP. Although I hate seeing new configurations, > I > > > > think > > > > > > this > > > > > > > is a good change. Combining these timeout behaviors into a > single > > > > > > > configuration was definitely a mistake, but we didn't > anticipate > > > the > > > > > > > problem with the producer id cache. I do wonder if we can make > > the > > > > > > default > > > > > > > a bit lower to reduce the chances that users will hit the same > > > memory > > > > > > > issues we have seen. After decoupling this configuration from > > > > > > > transactional.id.expiration.ms, the new timeout just needs to > > > cover > > > > > the > > > > > > > longest duration that a producer might be retrying the same > > Produce > > > > > > > request. 7 days seems too high. Although I think it could go a > > fair > > > > > even > > > > > > > lower, perhaps 1 day is a reasonable place to start? > > > > > > > > > > > > > > Thanks, > > > > > > > Jason > > > > > > > > > > > > > > On Mon, Jul 25, 2022 at 9:25 AM Justine Olshan > > > > > > > <jols...@confluent.io.invalid> > > > > > > > wrote: > > > > > > > > > > > > > > > Hey Bill, > > > > > > > > Thanks! I was just going to say that hopefully > > > > > > > > transactional.id.expiration.ms would also be over the > delivery > > > > > > timeout. > > > > > > > :) > > > > > > > > Thanks for the +1! > > > > > > > > > > > > > > > > Justine > > > > > > > > > > > > > > > > On Mon, Jul 25, 2022 at 9:17 AM Bill Bejeck < > bbej...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Justine, > > > > > > > > > > > > > > > > > > I just took another look at the KIP, and I realize my > > > > > > > question/suggestion > > > > > > > > > about default values has already been addressed in the > > > > > > `Compatibility` > > > > > > > > > section. > > > > > > > > > > > > > > > > > > I'm +1 on the KIP. > > > > > > > > > > > > > > > > > > -Bill > > > > > > > > > > > > > > > > > > On Thu, Jul 21, 2022 at 6:20 PM Bill Bejeck < > > bbej...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Justine, > > > > > > > > > > > > > > > > > > > > Thanks for the well written KIP, this looks like it will > > be a > > > > > > useful > > > > > > > > > > addition. > > > > > > > > > > > > > > > > > > > > Overall the KIP looks good to me, I have one > > > question/comment. > > > > > > > > > > > > > > > > > > > > You mentioned that setting the ` > producer.id.expiration.ms` > > > > less > > > > > > than > > > > > > > > the > > > > > > > > > > delivery timeout could lead to duplicates, which makes > > sense. > > > > To > > > > > > > help > > > > > > > > > > avoid this situation, do we want to consider a default > > value > > > > that > > > > > > is > > > > > > > > the > > > > > > > > > > same as the delivery timeout? > > > > > > > > > > > > > > > > > > > > Thanks again for the KIP. > > > > > > > > > > > > > > > > > > > > Bill > > > > > > > > > > > > > > > > > > > > On Thu, Jul 21, 2022 at 4:54 PM Justine Olshan > > > > > > > > > > <jols...@confluent.io.invalid> wrote: > > > > > > > > > > > > > > > > > > > >> Hey all! > > > > > > > > > >> > > > > > > > > > >> I'd like to start a discussion on my proposal to > separate > > > > > > time-based > > > > > > > > > >> producer ID expiration from transactional ID expiration > by > > > > > > > > introducing a > > > > > > > > > >> new configuration. > > > > > > > > > >> > > > > > > > > > >> The KIP Is pretty small and simple, but will be helpful > in > > > > > > > controlling > > > > > > > > > >> memory usage in brokers -- especially now that by > default > > > > > > producers > > > > > > > > are > > > > > > > > > >> idempotent and create producer ID state. > > > > > > > > > >> > > > > > > > > > >> Please take a look and leave any comments you may have! > > > > > > > > > >> > > > > > > > > > >> KIP: > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-854+Separate+configuration+for+producer+ID+expiry > > > > > > > > > >> JIRA: https://issues.apache.org/jira/browse/KAFKA-14097 > > > > > > > > > >> > > > > > > > > > >> Thanks! > > > > > > > > > >> Justine > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
