Howard, In 0.7, partitions are local to each broker. So, if you set # partitions to 3 per broker, you will get a total of 9 partitions with 3 brokers. Also, since there is no redundancy across brokers, if a broker goes down, none of all partitions on the broker is available. So num_partitions will shrink.
In 0.8, things are different. Partitions are at the cluster level. Number of partitions is determined at topic creation time and won't change as the # of brokers changes in the cluster. Also, 0.8 supports intra cluster replication. So a partition is still available if a single broker is down. Thanks, Jun On Fri, Oct 19, 2012 at 9:43 PM, howard chen <howac...@gmail.com> wrote: > Hi, > > Some basic questions concerning Partitioning & High availability > > Background: Assume I have 3 nodes of Kafka > > 1. Should I set the num.partitions to 3 or 1? > 2. By using the default DefaultPartitioner, it is > > hash(key)%num_partitions > > What if one of my node down (assume I am using 3 partitions)? > > a. All message sent to this node will be dropped? > b. Do I need to wait all the node come up again and able to consume > the message again? > > Thanks. >
