Would you also consider testing the different number of partitions to a failure point, based on increasing the load? I'd be interested to see where it tops out with different numbers of partitions. On Fri, 22 May 2015 at 6:48 am Carles Sistare <carles.sist...@googlemail.com> wrote:
> Thanks a lot guys for your answers, > We’ll be doing some benchmarks comparing different amount of partitions > for the same load. We’ll share the results. > > Cheers > > > On 21 May 2015, at 06:04, Saladi Naidu <naidusp2...@yahoo.com.INVALID> > wrote: > > > > In general partitions are to improve throughput by parallelism. From > your explanation below yes partitions are written to different physical > locations but still append only. With write ahead buffering and append only > writes, having partitions still will increase throughput. > > Below is an excellent article by Jun Rao about partitions and its impact > > How to choose the number of topics/partitions in a Kafka cluster? > > > > | | > > | | | | | | | | > > | How to choose the number of topics/partitions in a Kafka...This is a > common question asked by many Kafka users. The goal of this post is to > explain a few important determining factors and provide a few simple > formulas. More... | > > | | > > | View on blog.confluent.io | Preview by Yahoo | > > | | > > | | > > > > Naidu Saladi > > > > From: Daniel Compton <daniel.compton.li...@gmail.com> > > To: users@kafka.apache.org > > Sent: Wednesday, May 20, 2015 8:21 PM > > Subject: Re: Optimal number of partitions for topic > > > > One of the beautiful things about Kafka is that it uses the disk and OS > > disk caching really efficiently. > > > > Because Kafka writes messages to a contiguous log, it needs very little > > seek time to move the write head to the next point. Similarly for > reading, > > if the consumers are mostly up to date with the topic then the disk head > > will be close to the reading point, or the data will already be in disk > > cache and can be read from memory. Even cooler, because disk access is so > > predictable, the OS can easily prefetch data because it knows what you're > > going to ask for in 50ms. > > > > In summary, data locality rocks. > > > > What I haven't worked out is how the data locality benefits interact with > > having multiple partitions. I would assume that this would make things > > slower because you will be writing to multiple physical locations on > disk, > > though this may be ameliorated by only fsyncing every n seconds. > > > > I'd be keen to hear how this impacts it, or even better to see some > > benchmarks. > > > > -- > > Daniel. > > > > > > > > > > On Thu, 21 May 2015 at 12:56 pm Manoj Khangaonkar <khangaon...@gmail.com > > > > wrote: > > > >> With knowing the actual implementation details, I would get guess more > >> partitions implies more parallelism, more concurrency, more threads, > more > >> files to write to - all of which will contribute to more CPU load. > >> > >> Partitions allow you to scale by partitioning the topic across multiple > >> brokers. Partition is also a unit of replication ( 1 leader + replicas > ). > >> And for consumption of messages, the order is maintained within a > >> partitions. > >> > >> But if you put 100 partitions per topic on 1 single broker, I wonder if > it > >> is going to be an overhead. > >> > >> > >> > >> On Wed, May 20, 2015 at 1:02 AM, Carles Sistare <car...@ogury.co> > wrote: > >> > >>> Hi, > >>> We are implementing a Kafka cluster with 9 brokers into EC2 instances, > >> and > >>> we are trying to find out the optimal number of partitions for our > >> topics, > >>> finding out the maximal number in order not to update the partition > >> number > >>> anymore. > >>> What we understood is that the number of partitions shouldn’t affect > the > >>> CPU load of the brokers, but when we add 512 partitions instead of 128, > >> for > >>> instance, the CPU load exploses. > >>> We have three topics with 100000 messages/sec each, a replication > factor > >>> of 3 and two consumer groups for each partition. > >>> > >>> Could somebody explain, why the increase of the number of partitions > has > >> a > >>> so dramatic impact to the CPU load? > >>> > >>> > >>> Here under i paste the config file of kafka: > >>> > >>> broker.id=3 > >>> > >>> default.replication.factor=3 > >>> > >>> > >>> # The port the socket server listens on > >>> port=9092 > >>> > >>> # The number of threads handling network requests > >>> num.network.threads=2 > >>> > >>> # The number of threads doing disk I/O > >>> num.io.threads=8 > >>> > >>> # The send buffer (SO_SNDBUF) used by the socket server > >>> socket.send.buffer.bytes=1048576 > >>> > >>> # The receive buffer (SO_RCVBUF) used by the socket server > >>> socket.receive.buffer.bytes=1048576 > >>> > >>> # The maximum size of a request that the socket server will accept > >>> (protection against OOM) > >>> socket.request.max.bytes=104857600 > >>> > >>> > >>> > >>> # A comma seperated list of directories under which to store log files > >>> log.dirs=/mnt/kafka-logs > >>> > >>> # The default number of log partitions per topic. More partitions allow > >>> greater > >>> # parallelism for consumption, but this will also result in more files > >>> across > >>> # the brokers. > >>> num.partitions=16 > >>> > >>> # The minimum age of a log file to be eligible for deletion > >>> log.retention.hours=1 > >>> > >>> # The maximum size of a log segment file. When this size is reached a > new > >>> log segment will be created. > >>> log.segment.bytes=536870912 > >>> > >>> # The interval at which log segments are checked to see if they can be > >>> deleted according > >>> # to the retention policies > >>> log.retention.check.interval.ms=60000 > >>> > >>> # By default the log cleaner is disabled and the log retention policy > >> will > >>> default to just delete segments after their retention expires. > >>> # If log.cleaner.enable=true is set the cleaner will be enabled and > >>> individual logs can then be marked for log compaction. > >>> log.cleaner.enable=false > >>> > >>> # Timeout in ms for connecting to zookeeper > >>> zookeeper.connection.timeout.ms=1000000 > >>> > >>> auto.leader.rebalance.enable=true > >>> controlled.shutdown.enable=true > >>> > >>> > >>> Thanks in advance. > >>> > >>> > >>> > >>> Carles Sistare > >>> > >>> > >>> > >> > >> > >> -- > >> http://khangaonkar.blogspot.com/ > >> > > > >
