Kamal, Thank you. Using log.cleanup.policy=delete solved the issue.
-- IPVP From: Kamal C <kamaltar...@gmail.com><mailto:kamaltar...@gmail.com> Reply: users@kafka.apache.org <users@kafka.apache.org>><mailto:users@kafka.apache.org> Date: May 4, 2016 at 6:58:34 AM To: users@kafka.apache.org <users@kafka.apache.org>><mailto:users@kafka.apache.org> Subject: Re: What makes a message key mandatory and how to turn it off? Yes. Use *log.cleanup.policy=delete* if you don't want to compact topics. Reference: https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction On Wed, May 4, 2016 at 3:24 PM, I PVP <i...@hotmail.com> wrote: > Kamal, > > Could the log.cleanup.policy=compact on server<number>.properties be the > cause ? > > # /opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --topic > user_track --describe > Topic:user_track PartitionCount:1 ReplicationFactor:2 Configs: > Topic: user_track Partition: 0 Leader: 1003 Replicas: 1003,1002 Isr: > 1003,1002 > > > > Thanks > -- > IPVP > > > From: Kamal C <kamaltar...@gmail.com><mailto:kamaltar...@gmail.com> > Reply: users@kafka.apache.org <users@kafka.apache.org>><mailto: > users@kafka.apache.org> > Date: May 4, 2016 at 6:34:34 AM > To: users@kafka.apache.org <users@kafka.apache.org>><mailto: > users@kafka.apache.org> > Subject: Re: What makes a message key mandatory and how to turn it off? > > Can you describe your topic configuration using the below command ? > > *sh kafka-topics.sh --zookeeper localhost:2181 --topic <topic-name> > --describe* > > Key for a record is mandatory only for compacted topics. > > --Kamal > > On Wed, May 4, 2016 at 2:25 PM, I PVP <i...@hotmail.com> wrote: > > > HI all, > > > > What makes a message key mandatory and how to turn it off ? > > > > I am migrating the messaging piece of a java application from activemq to > > kafka. > > The application was publishing messages to kafka(0.9.0) with no issues > > while running on single broker on my dev machine. > > After turning it into multi-broker, by enabling 2 additional brokers, now > > I am seeing error messages on all producers saying that "Error when > sending > > message to topic <topic name> with key: null". The error happens on > > producing messages from Java client and for kafka-console-producer.sh > too. > > > > As a newbie I am not doing anything fancy around partitions, key and etc. > > > > The topics were being create with the following: > > /opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --topic <topic > > name> --create > > > > After going multi-broker I was forced to define partitions because the > > command above starting saying "Missing required argument "[partitions]” , > > so the all topics are being create with the following: > > > > /opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --topic <topic > > name> --create --replication-factor 2 -partitions 1 > > > > > > Messages were being sent by : > > ... > > ProducerRecord<String, Object> producerRecord = new > ProducerRecord<String, > > Object>(topicName, message); > > producer.send(producerRecord); > > ... > > > > To make things working again after the multi-broker change and the sunden > > mandatory message Key requirement I change the code to use a ramdonUUID > as > > the message key. > > ... > > ProducerRecord<String, Object> producerRecord = new > ProducerRecord<String, > > Object>(topicName,UUID.randomUUID().toString(), message); > > > > producer.send(producerRecord); > > ... > > > > After the change to multi-broker all topics were deleted, all log files > > were deleted , zookeeper entries were cleaned and topics were recreated. > > > > Even now working fine from the java client with the UUID as the message > > key it does not work from the command line, it looks like from the > > command line there is no way to set the message key and the command line > is > > critical when needed to do quick tests. > > > > If none logic at the applications consuming the messages requires a key > > why is it forcing to set a key ? The applications consuming the messages > > only need to consume the message as they come to the topic. > > > > Is there a way to turn off the mandatory message key on a multi-broker > > situation? > > > > Thanks > > > > -- > > IPVP > > > > >
