Hey, IIRC, Kafka does not provide any size related metadata. IMHO, there isn't any need of such metadata as well. If you want to process data only after you have certain amount of data to process, you can always consume records from Kafka into some buffer and process the buffered data once it is of required size. If your concern is what if your process, owning the buffer, crashes, how will you know where to start consuming from, Kafka lets you control when messages are considered consumed. As you are already using simple consumer, you have this flexibility. Hope this helps.
On Fri, Jul 17, 2015 at 2:21 PM, Hutashan Chandrakar <[email protected]> wrote: > Hello All, > > Requirement :- Process/Read (Consumer) data only if kafka message size > greater then specified size (configured). > > > I am using SimpleConsumer and trying to pull size wise (GB) message using > spark(kafka RDD).I am able to get earliest and latest offset using metadata > request, but not able to get any size related metadata information. > > > Is there way to get size of a message without fetchrequest in Kafka through > SimpleConsumer/highlevel api ? Is there any API/method to get size related > information in metadata request/response in kafka api. > > > > Thanks, > Hutashan Chandrakar > -- Regards, Ashish
