Hi, Tao Jiuming. The Diskless proposal doesn’t plan to modify the Kafka Storage API. I'm adding a note about this in the KIP. Please, see KIP-1163 for more details about the write path. We can also follow-up the discussion in its own discussion thread.
Thanks, Jorge. On Thu, 15 May 2025 at 12:57, Tao Jiuming <dao...@apache.org> wrote: > looks like it will write data to remote OSS directly, > I'm just wandering that do we need to make storage API async? > > Kafka's storage API is fully sync, in this case, if we write data to s3 > directly and wait the response, I'm afraid the throughput can be very low. > > But if we make it async, it will comes a lot of changes, threading model, > idempotence, txn, etc. > > I'm just curious about your consideration here. > > On 2025/04/16 11:58:22 Josep Prat wrote: > > Hi Kafka Devs! > > > > We want to start a new KIP discussion about introducing a new type of > > topics that would make use of Object Storage as the primary source of > > storage. However, as this KIP is big we decided to split it into multiple > > related KIPs. > > We have the motivational KIP-1150 ( > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics > ) > > that aims to discuss if Apache Kafka should aim to have this type of > > feature at all. This KIP doesn't go onto details on how to implement it. > > This follows the same approach used when we discussed KRaft. > > > > But as we know that it is sometimes really hard to discuss on that meta > > level, we also created several sub-kips (linked in KIP-1150) that offer > an > > implementation of this feature. > > > > We kindly ask you to use the proper DISCUSS threads for each type of > > concern and keep this one to discuss whether Apache Kafka wants to have > > this feature or not. > > > > Thanks in advance on behalf of all the authors of this KIP. > > > > ------------------ > > Josep Prat > > Open Source Engineering Director, Aiven > > josep.p...@aiven.io | +491715557497 | aiven.io > > Aiven Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > Anna Richardson, Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B > > >
