I just wonder if it is possible to read as batch using SimpleConsumer instead of HighLevel consumer? does same principle apply to low level consumer (i.e. SimpleConsumer)?
best, /Shahab On Tue, Aug 4, 2015 at 9:10 PM, Gwen Shapira <g...@confluent.io> wrote: > To add some internals, the high level consumer actually does read entire > batches from Kafka. It just exposes them to the user in an event loop, > because its a very natural API. Users can then batch events the way they > prefer. > > So if you are worried about batches being more efficient than single > events, you are covered! > > Gwen > > On Tue, Aug 4, 2015 at 12:04 PM, shahab <shahab.mok...@gmail.com> wrote: > > > Thanks a lot Shaminder for clarification and thanks Raja for pointing me > to > > the example. > > > > best, > > /shahab > > > > On Tue, Aug 4, 2015 at 6:06 PM, Rajasekar Elango <rela...@salesforce.com > > > > wrote: > > > > > Here is an example on what sharninder suggested > > > > > > > > > http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/ > > > > > > Thanks, > > > Raja. > > > > > > On Tue, Aug 4, 2015 at 12:01 PM, Sharninder <sharnin...@gmail.com> > > wrote: > > > > > > > You can't. Kafka is essentially a queue, so you always read messages > > one > > > > by one. What you can do is disable auto offset commit, read 100 > > messages, > > > > process them and then manually commit offset. > > > > > > > > -- > > > > Sharninder > > > > > > > > > On 04-Aug-2015, at 9:07 pm, shahab <shahab.mok...@gmail.com> > wrote: > > > > > > > > > > Hi, > > > > > > > > > > While we the producer can put data as batch in kafka server, I > > > couldn't > > > > > find any API (or any document) saying how we can fetch data as > batch > > > from > > > > > Kafka ? > > > > > Even when data is placed as batch in kafka server, still using High > > > Level > > > > > consumer I can only read one by one, and I can not specify. for > > > example, > > > > > read 100 items at once! > > > > > > > > > > Is this correct observation? or I am missing something? > > > > > > > > > > best, > > > > > /Shahab > > > > > > > > > > > > > > > > -- > > > Thanks, > > > Raja. > > > > > >
