[0.6] Consumer API Question - What if a ConsumerIterator has an error?

2011-12-22 Thread David Ross
Howdy, I have come across a potential issue with the Consumer API in Kafka 0.6. After some tests and reading through the code, I believe the following scenario could result in an unhappy situation: 1. Create 5 streams listening to a topic 2. One of the streams times out (or has some other e

Re: [0.6] Consumer API Question - What if a ConsumerIterator has an error?

2012-01-04 Thread David Ross
Another problem - just taking toString of an iterator in a failed state throws an exception. On Thu, Dec 22, 2011 at 1:40 AM, David Ross wrote: > Howdy, > > I have come across a potential issue with the Consumer API in Kafka 0.6. > After some tests and reading through the code, I

Thread Safety of KafkaStreams

2012-11-20 Thread David Ross
Hello, We want to process messages from a single KafkaStream in a number of processes. Is it possible to have this code executing in multiple threads against the same stream? for (message <- stream) { someBlockingOperation(message) } The scaladocs mention thread safety, but some of the code se

Re: Thread Safety of KafkaStreams

2012-11-21 Thread David Ross
arkhede > > > wrote: > > > David, > > > > One KafkaStream is meant to be iterated by a single thread. A better > > approach is to request higher number of streams > > from the Kafka consumer and let each process have its own KafkaStream. > > > > Tha