To use SimpleConsumer, you need to send TopicMetadataRequest (available in SimpleConsumer) to figure out the leader of each partition before making the fetch requests.
In both 0.7 and 0.8, a fetch request fetches data starting at the provided offset. In 0.8, offset is a sequential and evergrowing number (ie., 1, 2, 3, ...). Thanks, Jun On Wed, Nov 28, 2012 at 8:36 AM, Chris Curtin <curtin.ch...@gmail.com>wrote: > Hi, > > First, thanks for 0.8.0 I'm really impressed with the redundancy > and simplification of the producer and consumer models. > > I've upgraded my consumers from 0.7.2 to 0.8.0 and have some questions. > > I am using the Simple Consumer since I need to support replay of messages > at request from the client. > > In 0.7.2 I connected to each broker and requested messages for a specific > partition from all of them since data was being spread across all the > brokers. I see I don't need this in 0.8.0. Instead I need to know how to > find the primary Broker for a topic/partition. Is there an API for this? I > looked at the Zookeeper rebalance logic in the regular consumer, but was > wondering if that is somewhere that I can just call it instead of > duplicating an internal implementation. > > Before I figured out the 'primary' Broker I was able to connect to the > other brokers and ask for a topic/partition/offset. I got nothing back at > the client, but on the server there was an error about not being the > primary. Is there something on the client I can check to see if the Broker > I'm talking to is not the Primary (or no longer if something changes)? Why > doesn't the fetch logic return an error in this case? > > It looks like the meaning of the 'offset' passed to the fetch() method has > changed. in 0.7.2 passing an offset returned everything SINCE that offset, > with 0.8.0 the FetchRequest is everything STARTING with that offset. Am I > correct this changed? (or did I find a bug in my 0.7.2 logic?) > > Looks like the offset is now an sequential number with no gaps in 0.8.0 vs. > a byte offset in the file. Is this offset now guaranteed to be > unique/incrementing for a topic/partition, even if the primary changes? So > message 1000 on each of the replicas is the same? I used to keep a map of > Broker/offset in the client to know what data I'd already seen, but now I > THINK I can only keep the one offset. > > What is the purpose of the 'clientId' in the FetchRequestBuilder? For the > SimpleConsumer-based logic can this be anything? Can it be reused by > parallel connections, even if to the same topic/partition? > > Thanks, > > Chris >
