Re: Re: No key specified when sending the message to Kafka
Thank you,Ewen. I undertand it now. bit1...@163.com From: Ewen Cheslack-Postava Date: 2015-06-24 11:53 To: users@kafka.apache.org Subject: Re: No key specified when sending the message to Kafka It does balance data, but is sticky over short periods of time (for some definition of short...). See this FAQ for an explanation: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ? This behavior has been changed in the new producer to work the way you expected, and can be overridden by providing your own Partitioner interface. On Tue, Jun 23, 2015 at 8:28 PM, bit1...@163.com bit1...@163.com wrote: I have the following code snippet that use Kafka Producer to send message(No key is specified in the KeyedMessage): val data = new KeyedMessage[String, String](topicName, msg); Kafka_Producer.send(data) Kafka_Producer is an instance of kafka.producer.Producer. With above code, I observed that the message sent to kafka is not partitioned(That is, all the messages are pushed to partition 0). If I give the message a key, they it can be partitioned across the topic. So, my question is: If no key is provided in the message , will Kafka producer not automatically partition the message with some built-in balancing algorithm? Thanks bit1...@163.com -- Thanks, Ewen
No key specified when sending the message to Kafka
I have the following code snippet that use Kafka Producer to send message(No key is specified in the KeyedMessage): val data = new KeyedMessage[String, String](topicName, msg); Kafka_Producer.send(data) Kafka_Producer is an instance of kafka.producer.Producer. With above code, I observed that the message sent to kafka is not partitioned(That is, all the messages are pushed to partition 0). If I give the message a key, they it can be partitioned across the topic. So, my question is: If no key is provided in the message , will Kafka producer not automatically partition the message with some built-in balancing algorithm? Thanks bit1...@163.com
Re: No key specified when sending the message to Kafka
It does balance data, but is sticky over short periods of time (for some definition of short...). See this FAQ for an explanation: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ? This behavior has been changed in the new producer to work the way you expected, and can be overridden by providing your own Partitioner interface. On Tue, Jun 23, 2015 at 8:28 PM, bit1...@163.com bit1...@163.com wrote: I have the following code snippet that use Kafka Producer to send message(No key is specified in the KeyedMessage): val data = new KeyedMessage[String, String](topicName, msg); Kafka_Producer.send(data) Kafka_Producer is an instance of kafka.producer.Producer. With above code, I observed that the message sent to kafka is not partitioned(That is, all the messages are pushed to partition 0). If I give the message a key, they it can be partitioned across the topic. So, my question is: If no key is provided in the message , will Kafka producer not automatically partition the message with some built-in balancing algorithm? Thanks bit1...@163.com -- Thanks, Ewen
