Hi Tom, Thanks for adding this performance test results. So basically, for acks=1 case, it's what we expected.
I have some comments about it, I'll reply in the discussion thread in KIP-1176. Thank you. Luke On Thu, Aug 28, 2025 at 4:53 AM Thomas Thornton <tthorn...@salesforce.com.invalid> wrote: > Hi Luke, > > Thanks for creating this discussion on these KIPs. > > I'm collaborating with Henry on KIP-1176. We have deployed the Kafka fork > to one of our Kafka clusters. We collected performance data (full results > here > < > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=354454090#KIP1176:TieredStorageforActiveLogSegment-Appendix:PerformanceData > >) > for the acks=1 case. Overall we see comparable performance (throughput & > latency) for the producer path between a standard topic and one with > KIP-1176 code. Consumer latency is slightly higher as data must travel an > additional hop to/from S3E1Z. There are additional tunings we are aware of > that could further boost performance (e.g., tuning the interval that the > leader/follower remote WAL tasks read/write data to/from cloud storage). > > Are there any other benchmarks the community would find useful? Any > questions on these findings? > > Thanks, > Tom > > On Tue, Aug 5, 2025 at 1:31 AM Luke Chen <show...@gmail.com> wrote: > > > Hi all, > > > > The Kafka community is currently seeing an unprecedented situation with > > three KIPs (KIP-1150, IP-1176, KIP-1183) simultaneously addressing the > same > > challenge of high replication costs when running Kafka across multiple > > cloud availability zones. Each KIP offers a different solution to this > > issue. While diversity of innovative ideas is a key strength of > open-source > > projects, it creates a burden for reviewers and users who must compare > and > > comment on multiple proposals simultaneously. Furthermore, discussion > > around the three KIPs has stalled for over two months now. This could be > > due to the authors being hesitant to proceed due to the existence of > > alternative, potentially conflicting, solutions. Addressing replication > > cost is a key concern of Kafka’s userbase and we should try to move the > > conversation forward if we can. > > > > From what I understand, these three KIPs are not mutually exclusive. But > > adopting all three KIPs in the community might not be what we expect. > Thus, > > I would like to *start a discussion on how we could move the conversation > > forward*. > > > > To save time for the KIP readers/reviewers, I have created this document > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/The+Path+Forward+for+Saving+Cross-AZ+Replication+Costs+KIPs > > >[1] > > to help summarize each of the KIPs and describe their current status. > *Hope > > to get some suggestions/feedback from the community*. > > > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/KAFKA/The+Path+Forward+for+Saving+Cross-AZ+Replication+Costs+KIPs > > > > KIP-1150: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics > > KIP-1176 > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+TopicsKIP-1176 > > > > : > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1176%3A+Tiered+Storage+for+Active+Log+Segment > > KIP-1183 > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1176%3A+Tiered+Storage+for+Active+Log+SegmentKIP-1183 > > > > : > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1183%3A+Unified+Shared+Storage > > > > > > Thank you. > > Luke > > >