Re: [DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted

Hello Stanislav, Thanks again for your comments. I understand, and Im happy to hear that my usecase is rare. The reason is that before going to real production, we are forced to build a prototype, with limited resources, but still resilient enough to pass acceptance tests. I agree the whitelist

Re: [DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted

Hey Jose, Thanks for sharing your use cases. >From my experience, it is uncommon to run with a retention.ms setting small enough that it can make you lose messages when your consumers can't catch up. If you are concerned with data loss, I think the cost investment into hardware is generally worth

Re: [DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted

Hello, Thanks for taking the time to review my KIP! I will describe some production scenarios I faced to better explain the reasons for this KIP. * Usecase 1: batch processing of files. A batch is producing huge files that must be processed. Each line of the file will be a message produced to a

Re: [DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted

Hey Jose, Thanks for the KIP. I think that Colin was referring to an existing client metric called "kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{client-id}",name=records-lag-max", exposed on the consumer application. You should be able to use that to get a sense of how far

Re: [DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted

Hello Kamal, The compacted topics are excluded from the KIP, because users of compacted topics are mainly interested on the last state for a certain key, and can afford to miss intermediary states. Technically is possible to know if the topic is compacted through "log.config.compact" attribute.

Re: [DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted

Jose, How do you differentiate the compaction topics from the time retention topics? Deleting a message due to compaction policy is a valid case and users won't be interested in monitoring/reading those deleted messages. Thanks, Kamal On Tue, Jul 23, 2019 at 4:00 AM Jose M wrote: > Hi

Re: [DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted

Hi Colin, Thanks a lot for your feedback. Please note that I only propose to log when a message is lost this for a set of consumer groups, not as default behaviour for all consumer groups. But in fact, I agree with you that to log a line per message expired can be quite lot, and that is not the

Re: [DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted

Hi Jose, One issue that I see here is that the number of log messages could be huge. I've seen people create tens of thousands of consumer groups. People can also have settings that create pretty small log files. A message per log file per group could be quite a lot of messages. A log

Re: [DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted

Hello, I didn't get any feedback on this small KIP-490 . In summary, I propose a way to be noticed when messages are being removed due to retention policy, without

[DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted

Hello, I would like to know what do you think on KIP-490: https://cwiki.apache.org/confluence/display/KAFKA/KIP-490%3A+log+when+consumer+groups+lose+a+message+because+offset+has+been+deleted