Hi all, Thank you all for your questions and design input on KIP-1150.
We have just updated KIP-1150 and KIP-1163 with a new design. To summarize the changes: 1. The design prioritizes integrating with the existing KIP-405 Tiered Storage interfaces, permitting data produced to a Diskless topic to be moved to tiered storage. This lowers the scalability requirements for the Batch Coordinator component, and allows Diskless to compose with Tiered Storage plugin features such as encryption and alternative data formats. 2. Consumer fetches are now served from local segments, making use of the indexes, page cache, request purgatory, and zero-copy functionality already built into classic topics. However, local segments are now considered cache elements, do not need to be durably stored, and can be built without contacting any other replicas. 3. The design has been simplified substantially, by removing the previous Diskless consume flow, distributed cache component, and "object compaction/merging" step. The design maintains leaderless produces as enabled by the Batch Coordinator, and the same latency profiles as the earlier design, while being simpler and integrating better into the existing ecosystem. Thanks, and we are eager to hear your feedback on the new design. Greg Harris On Mon, Jul 21, 2025 at 3:30 PM Jun Rao <j...@confluent.io.invalid> wrote: > Hi, Jan, > > For me, the main gap of KIP-1150 is the support of all existing client > APIs. Currently, there is no design for supporting APIs like transactions > and queues. > > Thanks, > > Jun > > On Mon, Jul 21, 2025 at 3:53 AM Jan Siekierski > <jan.siekier...@kentra.io.invalid> wrote: > > > Would it be a good time to ask for the current status of this KIP? I > > haven't seen much activity here for the past 2 months, the vote got > vetoed > > but I think the pending questions have been answered since then. KIP-1183 > > (AutoMQ's proposal) also didn't have any activity since May. > > > > In my eyes KIP-1150 and KIP-1183 are two real choices that can be > > made, with a coordinator-based approach being by far the dominant one > when > > it comes to market adoption - but all these are standalone products. > > > > I'm a big fan of both approaches, but would hate to see a stall. So the > > question is: can we get an update? > > > > Maybe it's time to start another vote? Colin McCabe - have your questions > > been answered? If not, is there anything I can do to help? I'm deeply > > familiar with both architectures and have written about both? > > > > Kind regards, > > Jan > > > > On Tue, Jun 24, 2025 at 10:42 AM Stanislav Kozlovski < > > stanislavkozlov...@apache.org> wrote: > > > > > I have some nits - it may be useful to > > > > > > a) group all the KIP email threads in the main one (just a bunch of > links > > > to everything) > > > b) create the email threads > > > > > > It's a bit hard to track it all - for example, I was searching for a > > > discuss thread for KIP-1165 for a while; As far as I can tell, it > doesn't > > > exist yet. > > > > > > Since the KIPs are published (by virtue of having the root KIP be > > > published, having a DISCUSS thread and links to sub-KIPs where were > aimed > > > to move the discussion towards), I think it would be good to create > > DISCUSS > > > threads for them all. > > > > > > Best, > > > Stan > > > > > > On 2025/04/16 11:58:22 Josep Prat wrote: > > > > Hi Kafka Devs! > > > > > > > > We want to start a new KIP discussion about introducing a new type of > > > > topics that would make use of Object Storage as the primary source of > > > > storage. However, as this KIP is big we decided to split it into > > multiple > > > > related KIPs. > > > > We have the motivational KIP-1150 ( > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics > > > ) > > > > that aims to discuss if Apache Kafka should aim to have this type of > > > > feature at all. This KIP doesn't go onto details on how to implement > > it. > > > > This follows the same approach used when we discussed KRaft. > > > > > > > > But as we know that it is sometimes really hard to discuss on that > meta > > > > level, we also created several sub-kips (linked in KIP-1150) that > offer > > > an > > > > implementation of this feature. > > > > > > > > We kindly ask you to use the proper DISCUSS threads for each type of > > > > concern and keep this one to discuss whether Apache Kafka wants to > have > > > > this feature or not. > > > > > > > > Thanks in advance on behalf of all the authors of this KIP. > > > > > > > > ------------------ > > > > Josep Prat > > > > Open Source Engineering Director, Aiven > > > > josep.p...@aiven.io | +491715557497 | aiven.io > > > > Aiven Deutschland GmbH > > > > Alexanderufer 3-7, 10117 Berlin > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > > > Anna Richardson, Kenneth Chen > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > >
