Just note that this is not a universal solution. Many use-cases care about which partition you end up writing to since partitions are used to... well, partition logical entities such as customers and users.
On Wed, Oct 15, 2014 at 9:03 PM, Jun Rao <jun...@gmail.com> wrote: > Kyle, > > What you wanted is not supported out of box. You can achieve this using the > new java producer. The new java producer allows you to pick an arbitrary > partition when sending a message. If you receive NotEnoughReplicasException > when sending a message, you can resend it to another partition. > > Thanks, > > Jun > > On Tue, Oct 14, 2014 at 1:51 PM, Kyle Banker <kyleban...@gmail.com> wrote: > >> Consider a 12-node Kafka cluster with a 200-parition topic having a >> replication factor of 3. Let's assume, in addition, that we're running >> Kafka v0.8.2, we've disabled unclean leader election, acks is -1, and >> min.isr is 2. >> >> Now suppose we lose 2 nodes. In this case, there's a good chance that 2/3 >> replicas of one or more partitions will be unavailable. This means that >> messages assigned to those partitions will not be writable. If we're >> writing a large number of messages, I would expect that all producers would >> eventually halt. It is somewhat surprising that, if we rely on a basic >> durability setting, the cluster would likely be unavailable even after >> losing only 2 / 12 nodes. >> >> It might be useful in this scenario for the producer to be able to detect >> which partitions are no longer available and reroute messages that would >> have hashed to the unavailable partitions (as defined by our acks and >> min.isr settings). This way, the cluster as a whole would remain available >> for writes at the cost of a slightly higher load on the remaining machines. >> >> Is this limitation accurately described? Is the proposed producer >> functionality worth pursuing? >>
