Re: Can anyone help me to send messages in their original order?

2018-05-26 Thread Damian Guy
Hi Raymond, If you want all messages delivered in order then you should create the topic with 1 partition. If you want ordering guarantees for messages with the same key, then you need to produce the messages with a key. Using the console producer you can do that by adding --property

Re: Can anyone help me to send messages in their original order?

2018-05-26 Thread Raymond Xie
Thank you so much Hans for your enlightening, it is definitely greatly helpful to me as a new starter. So for my case, what is the right options I should put together to run the commands for producer and consumer respectively? Thanks. **

Re: Can anyone help me to send messages in their original order?

2018-05-26 Thread Hans Jespersen
There are two concepts in Kafka that are not always familiar to people who have used other pub/sub systems. 1) partitions: Kafka topics are partitioned which means a single topic is sharded into multiple pieces that are distributed across multiple brokers in the cluster for parallel

Re: Can anyone help me to send messages in their original order?

2018-05-26 Thread Raymond Xie
Thanks. By default, can you explain me why I received the message in wrong order? Note there are only 9 lines from 1 to 9, but on consumer side their original order becomes messed up. ~~~sent from my cell phone, sorry if there is any typo Hans Jespersen 于 2018年5月26日周六

Re: Can anyone help me to send messages in their original order?

2018-05-25 Thread Hans Jespersen
If you create a topic with one partition they will be in order. Alternatively if you publish with the same key for every message they will be in the same order even if your topic has more than 1 partition. Either way above will work for Kafka. -hans > On May 25, 2018, at 8:56 PM, Raymond Xie

Can anyone help me to send messages in their original order?

2018-05-25 Thread Raymond Xie
Hello, I just started learning Kafka and have the environment setup on my hortonworks sandbox at home vmware. test.csv is what I want the producer to send out: more test1.csv ./kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic kafka-topic2 1, abc 2, def ... 8, vwx 9,