Hi Luke, Thanks for creating this discussion on these KIPs.
I'm collaborating with Henry on KIP-1176. We have deployed the Kafka fork to one of our Kafka clusters. We collected performance data (full results here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=354454090#KIP1176:TieredStorageforActiveLogSegment-Appendix:PerformanceData>) for the acks=1 case. Overall we see comparable performance (throughput & latency) for the producer path between a standard topic and one with KIP-1176 code. Consumer latency is slightly higher as data must travel an additional hop to/from S3E1Z. There are additional tunings we are aware of that could further boost performance (e.g., tuning the interval that the leader/follower remote WAL tasks read/write data to/from cloud storage). Are there any other benchmarks the community would find useful? Any questions on these findings? Thanks, Tom On Tue, Aug 5, 2025 at 1:31 AM Luke Chen <show...@gmail.com> wrote: > Hi all, > > The Kafka community is currently seeing an unprecedented situation with > three KIPs (KIP-1150, IP-1176, KIP-1183) simultaneously addressing the same > challenge of high replication costs when running Kafka across multiple > cloud availability zones. Each KIP offers a different solution to this > issue. While diversity of innovative ideas is a key strength of open-source > projects, it creates a burden for reviewers and users who must compare and > comment on multiple proposals simultaneously. Furthermore, discussion > around the three KIPs has stalled for over two months now. This could be > due to the authors being hesitant to proceed due to the existence of > alternative, potentially conflicting, solutions. Addressing replication > cost is a key concern of Kafka’s userbase and we should try to move the > conversation forward if we can. > > From what I understand, these three KIPs are not mutually exclusive. But > adopting all three KIPs in the community might not be what we expect. Thus, > I would like to *start a discussion on how we could move the conversation > forward*. > > To save time for the KIP readers/reviewers, I have created this document > < > https://cwiki.apache.org/confluence/display/KAFKA/The+Path+Forward+for+Saving+Cross-AZ+Replication+Costs+KIPs > >[1] > to help summarize each of the KIPs and describe their current status. *Hope > to get some suggestions/feedback from the community*. > > > [1] > > https://cwiki.apache.org/confluence/display/KAFKA/The+Path+Forward+for+Saving+Cross-AZ+Replication+Costs+KIPs > > KIP-1150: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics > KIP-1176 > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+TopicsKIP-1176> > : > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1176%3A+Tiered+Storage+for+Active+Log+Segment > KIP-1183 > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1176%3A+Tiered+Storage+for+Active+Log+SegmentKIP-1183> > : > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1183%3A+Unified+Shared+Storage > > > Thank you. > Luke >
