Hi Henry, Thanks for the KIP. This is quite interesting and it fills a use-case that many people ask about.
One thing we've done at LittleHorse is use a simple Kafka Streams topology to simulate delayed production. We store messages in a state store until their maturation time and then forward them onwards from a punctuator. Overall, it's about 20 lines of code total to do this, and it works great. The advantages of your KIP over that simple solution are that a user of 1277 wouldn't have to worry about managing a Streams topology (which is...hard sometimes). On the other hand, the implementation seems like a heavy lift, especially when reconciling multiple batches when tiered storage is enabled. Is it worth the additional effort? Colt McNealy Founder, www.littlehorse.io On 2026/01/30 07:34:09 Henry Haiying Cai via dev wrote: > Hello Kafka Developers, > > I would like to start discussing KIP-1277: Support Delayed Message in Kafka. > > A common queue scheduling feature is delayed messages where the message is not supposed to be delivered or consumed right away. The use case is a large influx of messages or other activities happening in the system at the moment, the message producer wants to make sure the messages are being consumed/processed in a little bit later time or the message consumption is being spread over a period of time. Another common use is message retry handling (e.g. retries in the new Kafka Queue feature), when a message consumer/worker cannot process the message due to some transient failures in external systems, usually the worker wants to unacknowledge the message and retry it later. The retry is usually more ideal to be scheduled at a later time usually with some exponential backoff time interval. > > Since Kafka was lacking support for message scheduling or delayed message delivery, users have turned to other queuing systems for these features. For example, users have been using AWS SQS delayed message / delayed topic to deliver messages later within a short time frame (e.g. within 15 minutes) and use DynamoDB or traditional database tables for the delayed message for longer delayed duration. > > We are proposing to implement delayed messages for Kafka to fill in this feature gap. Similar to SQS, the proposal is also focusing on messages delayed delivery for a short time window (up to 15 minutes late). In our use cases, most of the message late delivery is within 1 minute. > > KIP-1277 can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1277%3A+Support+Delayed+Message+in+Kafka > > Looking forward to suggestions and feedback :) > > Best, > Henry Cai and Tom Thornton >
