Hi Hisham, > I've whipped up a quick Scala example of an embedded Kafka broker, > producer, and consumer. I don't use ZK and I use the simple consumer > (manual offset management). > > https://gist.github.com/2650743 > > Hopefully this helps clear things up a bit.
Yes, thanks - and sorry for the delay in responding, I had to step away from this project for a bit. I was easily able to get the equivalent Java code running. The remaining question I've got now is how to cleanly shut things down. I can terminate the Kafka server, but I can't terminate the threads that are running as consumers. Specifically when I terminate the executor that I'm using to run the consumer threads, this fails, and in the logs I see: [2012-06-27 20:07:29,696] INFO fetch reconnect due to java.nio.channels.ClosedByInterruptException (kafka.consumer.SimpleConsumer) The consumer threads are blocked on SimpleConsumer#fetch, and the fetch reconnect means that this doesn't ever return. Any input on how best to terminate properly in a test environment? Thanks! -- Ken > On Wed, May 9, 2012 at 9:14 PM, Ken Krugler <kkrugler_li...@transpac.com> > wrote: >> Hi Hisham, >> >> Thanks for chiming in - see below. >> >>> When you call shutdown() on KafkaServerStartable you should also call >>> awaitShutdown if you're expecting your app / JVM to terminate (perhaps >>> you are not). >> >> It's been shutting down properly, but you're right that calling >> awaitShutdown() ensures that everything has been properly cleaned up before >> I continue. >> >>> Also take into account that if the embedded KafkaServer >>> instance in KafkaServerStartable throws an exception on shutdown it >>> will call Runtime.getRuntime.halt(1) (not sure if you want this >>> behaviour). >> >> Not really, but hopefully my unit tests aren't triggering exceptions in the >> KafkaServer :) >> >>> I'm embedding a Kafka broker and producer in an experimental lib doing >>> pretty much what you're doing except I use KafkaServer directly and >>> make no use of ZK. >> >> I initially tried that, but ran into the issue where Kafka consumers always >> create a ZooKeeper client, which expects to have a ZooKeeper server running. >> >> Do you know if it's possible to set up a complete Kafka environment (broker, >> producer, consumer) without ZooKeeper? >> >> -- Ken >> >> -------------------------- >> Ken Krugler >> http://www.scaleunlimited.com >> custom big data solutions & training >> Hadoop, Cascading, Mahout & Solr >> >> >> >> > > > > -- > Hisham Mardam-Bey > [ Director of Engineering ] [ Mate1 Inc. ] > > A: Because it messes up the order in which people normally read text. > Q: Why is top-posting such a bad thing? > A: Top-posting. > Q: What is the most annoying thing in e-mail? > > -=[ Codito Ergo Sum ]=- -------------------------- Ken Krugler http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Mahout & Solr
