Hi, Yulin, I assume the performance issue that you mentioned is for writes instead of reads. By default, Kafka flushes data to disks asynchronously in batches. So, even when there are multiple files to write, the batching can amortize the HDD seek overhead. It would be useful to understand the number of I/Os and the size of each I/O in your environment.
Thanks, Jun On Fri, Nov 2, 2018 at 9:50 PM, Yuanjin Lin <linyuan...@zhihu.com> wrote: > Colin, Thanks for the meaningful reply! > > We are 100% sure those HDDs are the bottleneck. Almost 90% alerts are about > HDDs. I am the guy who have to deal with it. The common scenario would be > 100-400 partitions per HDD(2TB size). Due to some historical reasons, > developers in my company tend to put everything to Kafka, cuz it makes them > feel safe. Although, we have over 2k servers for Kafka. I still can receive > alerts everyday. > > If the modification I proposed is not too hard, I will begin to do it next > month. > > > On Sat, Nov 3, 2018 at 1:36 AM Colin McCabe <cmcc...@apache.org> wrote: > > > On Fri, Nov 2, 2018, at 03:14, Yuanjin Lin wrote: > > > Hi all, > > > > > > I am a software engineer from Zhihu.com. Kafka is so great and used > > heavily > > > in Zhihu. There are probably over 2K Kafka brokers in total. > > > > > > However, we are suffering from the problem that the performance > degrades > > > rapidly when the number of topics increases(sadly, we are using HDD). > > > > Hi Yuanjin, > > > > How many partitions are you trying to create? > > > > Do you have benchmarks confirming that disk I/O is your bottleneck? > There > > are a few cases where large numbers of partitions may impose CPU and > > garbage collection burdens. The patch on > > https://github.com/apache/kafka/pull/5206 illustrates one of them. > > > > > We are considering separating the logic layer and the storage layer of > > Kafka > > > broker like Apache Pulsar. > > > > > > After the modification, a server may have several Kafka brokers and > more > > > topics. Those brokers all connect to a sole storage engine via RP The > > > sole storage can do the load balancing work easily, and avoid creating > > too > > > many files which hurts HDD. > > > > > > Is it hard? I think replacing the stuff in `Kafka.Log` would be enough, > > > right? > > > > It would help to know what the problem is here. If the problem is a > large > > number of files, then maybe the simplest approach would be creating fewer > > files. You don't need to introduce a new layer of servers in order to do > > that. You could use something like RocksDB to store messages and > indices, > > or create your own file format which combined together things which were > > previously separate. For example, we could combine the timeindex and > index > > files. > > > > As I understand it, Pulsar made the decision to combine together data > from > > multiple partitions in a single file. Sometimes a very large number of > > partitions. This is great for writing, but not so good if you want to > read > > historical data from a single topic. > > > > regards, > > Colin > > >
