Thanks! Will let this list know if and when I run a log.dirs vs. num.io.threads test.
________________________________ Hah :) I think this deserves an experiment. I’d try setting up some tests with one, two, four, and eight log directories per disk and running some performance tests. I’d be interested to see your results. > On Mar 11, 2020, at 5:45 PM, Eugen Dueck <eu...@tworks.co.jp> wrote: > > I'm asking the questions here! 🙂 > So is that the way to tune the broker if it does not achieve disk throughput? > > ________________________________ > 差出人: Peter Bukowinski <pmb...@gmail.com> > 送信日時: 2020年3月12日 9:38 > > Couldn’t the same be accomplished by increasing the num.io.threads broker > setting? > >> On Mar 11, 2020, at 5:15 PM, Eugen Dueck <eu...@tworks.co.jp> wrote: >> >> So there is not e.g. a single thread responsible per directory in log.dirs >> that could become a bottleneck relative to SSD throughput of GB/s? >> >> This is in fact the case for Apache Pulsar, and the openmessaging benchmark >> uses 4 directories on the same SSD to increase throughput. >> >> ________________________________ >> 差出人: Peter Bukowinski <pmb...@gmail.com> >> 送信日時: 2020年3月12日 8:51 >> >>> On Mar 11, 2020, at 4:28 PM, Eugen Dueck <eu...@tworks.co.jp> wrote: >>> >>> So log.dirs should contain only one entry per HDD disk, to avoid random >>> seeks. >>> What about SSDs? Can throughput be increased by specifying multiple >>> directories on the same SSD? >> >> >> Given a constant number of partitions, I don’t see any advantage to >> splitting partitions among multiple log directories vs. keeping them all in >> one (per disk). You’d still have the same total number of topic-partition >> directories and the same number of topic-partition leaders. >> >> If you want to increase throughput, focus on using the appropriate number of >> partitions. >> >> — >> Peter Bukowinski >
