There are two concepts in Kafka that are not always familiar to people who have used other pub/sub systems.
1) partitions: Kafka topics are partitioned which means a single topic is sharded into multiple pieces that are distributed across multiple brokers in the cluster for parallel processing. Order is guaranteed per partition (not per topic). You can think of each kafka topic partition like an exclusive queue is traditional messaging systems and order is not guaranteed when the data is spread out across multiple queues in tradition messaging either. 2) keys Kafka messages have keys in addition the value (I.e body) and the header. When messages are published with the same key they will be all be sent in order to the same partition. If messages are published with a “null” key then they will be spread out round robin across all partitions (which is what you have done). Conclusion You will see ordered delivery if your either use a key when you publish or create a topic with one partition. -hans > On May 26, 2018, at 7:59 AM, Raymond Xie <xie3208...@gmail.com> wrote: > > Thanks. By default, can you explain me why I received the message in wrong > order? Note there are only 9 lines from 1 to 9, but on consumer side their > original order becomes messed up. > > ~~~sent from my cell phone, sorry if there is any typo > > Hans Jespersen <h...@confluent.io> 于 2018年5月26日周六 上午12:16写道： >> If you create a topic with one partition they will be in order. >> >> Alternatively if you publish with the same key for every message they will >> be in the same order even if your topic has more than 1 partition. >> >> Either way above will work for Kafka. >> >> -hans >> >> > On May 25, 2018, at 8:56 PM, Raymond Xie <xie3208...@gmail.com> wrote: >> > >> > Hello, >> > >> > I just started learning Kafka and have the environment setup on my >> > hortonworks sandbox at home vmware. >> > >> > test.csv is what I want the producer to send out: >> > >> > more test1.csv ./kafka-console-producer.sh --broker-list >> > sandbox.hortonworks.com:6667 --topic kafka-topic2 >> > >> > 1, abc >> > 2, def >> > ... >> > 8, vwx >> > 9, zzz >> > >> > What I received are all the content of test.csv, however, not in their >> > original order; >> > >> > kafka-console-consumer.sh --zookeeper 192.168.112.129:2181 --topic >> > kafka-topic2 >> > >> > 2, def >> > 1, abc >> > ... >> > 9, zzz >> > 8, vwx >> > >> > >> > I read from google that partition could be the feasible solution, however, >> > my questions are: >> > >> > 1. for small files like this one, shall I really do the partitioning? how >> > small a partition would be acceptable to ensure the sequence? >> > 2. for big files, each partition could still contain multiple lines, how to >> > ensure all the lines in each partition won't get messed up on consumer >> > side? >> > >> > >> > I also want to know what is the best practice to process large volume of >> > data through kafka? There should be better way other than console command. >> > >> > Thank you very much. >> > >> > >> > >> > *------------------------------------------------* >> > *Sincerely yours,* >> > >> > >> > *Raymond*
