I am trying to write a junit test for working with an embedded Kafka 0.8 server where I send and receive messages just to verify my embedded server, producer and consumer wrappers work. Order of operation in the junit looks something like:
-Start zk. [own thread] (wait for init) -Start Kafka [own thread] (wait for init) -Start consumer [own thread] (I block on my threads run method releasing a lock once it starts). consumer is copied from: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example -Start producer (dont see anything to wait/block on, so I move along) Now that things are online, I send 5 messages out from the producer. There exists no topic, so one is created auto-magically by kafka (btw, why is there no api to create topics in java?) I then read from the consumer and compare how many I received against the sent ones. What I notice is that when I run my test over and over, I will get anywhere from 0-5 messages received. The logs look like for some reason the client seems to re-establish its connection when messages start being sent and depending on how long that takes depends on how many of the messages I'll get to read. Is the high level consumer meant to be used this way? If a high level consumer is started first am I not guaranteed all messages produced from that point on? I have a single stream asked for from the topic so according to the doc that single stream should get all messages on the topic (which inits with 2 partitions). Does anyone have any more insight or should I just migrate into the simple consumer example? ~Garrett
