Hi, First, thanks for 0.8.0 I'm really impressed with the redundancy and simplification of the producer and consumer models.
I've upgraded my consumers from 0.7.2 to 0.8.0 and have some questions. I am using the Simple Consumer since I need to support replay of messages at request from the client. In 0.7.2 I connected to each broker and requested messages for a specific partition from all of them since data was being spread across all the brokers. I see I don't need this in 0.8.0. Instead I need to know how to find the primary Broker for a topic/partition. Is there an API for this? I looked at the Zookeeper rebalance logic in the regular consumer, but was wondering if that is somewhere that I can just call it instead of duplicating an internal implementation. Before I figured out the 'primary' Broker I was able to connect to the other brokers and ask for a topic/partition/offset. I got nothing back at the client, but on the server there was an error about not being the primary. Is there something on the client I can check to see if the Broker I'm talking to is not the Primary (or no longer if something changes)? Why doesn't the fetch logic return an error in this case? It looks like the meaning of the 'offset' passed to the fetch() method has changed. in 0.7.2 passing an offset returned everything SINCE that offset, with 0.8.0 the FetchRequest is everything STARTING with that offset. Am I correct this changed? (or did I find a bug in my 0.7.2 logic?) Looks like the offset is now an sequential number with no gaps in 0.8.0 vs. a byte offset in the file. Is this offset now guaranteed to be unique/incrementing for a topic/partition, even if the primary changes? So message 1000 on each of the replicas is the same? I used to keep a map of Broker/offset in the client to know what data I'd already seen, but now I THINK I can only keep the one offset. What is the purpose of the 'clientId' in the FetchRequestBuilder? For the SimpleConsumer-based logic can this be anything? Can it be reused by parallel connections, even if to the same topic/partition? Thanks, Chris
