What are you going to do with the messages every 15 minutes? One way I can think of is to have two consumers in your application. One of them reads messages and just keeps track of the offsets for each hour. The other consumer then uses this info to pull the data. You could publish these offsets to a topic keyed on something like yyyy-mm-dd-hh24 or keep them in memory ( if running in the same application ). You would need some one-time process to create the offsets for the first time.
-Dave -----Original Message----- From: Kaustuv Bhattacharya [mailto:kaustuvl...@gmail.com] Sent: Sunday, September 3, 2017 1:08 PM To: users@kafka.apache.org Subject: Time based data retrieval from Kafka Topic *Hi All,* *I need a solution to the below Problem statement -* *How to retrieve only last 1 hour data from an existing Kafka Topic, on 1st & every consecutive (at every 15 mins interval) of the client application?* *Note:- The existing Topic is accumulating data since last 6 months.* *Regards,* *Kaustuv Bhattacharya* This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
