Yup, exactly as Kant said. Also make sure that the retention of the offsets topic is an upper bound across all topics. So in this case, don't create any other topics in the future with retention of more than 30 days or otherwise they may have the same problem too.
On May 21, 2017 03:25, "Abhimanyu Nagrath" <abhimanyunagr...@gmail.com> wrote: > Hi Kant, > > Thanks for the suggestion. > > > Regards, > Abhimanyu > > On Sun, May 21, 2017 at 3:44 PM, kant kodali <kanth...@gmail.com> wrote: > >> @Abhimanyu You can try setting offset.retention = 30 (log.retention). At >> most, you will have a storage overhead of 5 million msgs per day * 30 >> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you have a >> TB of hard disk) >> >> On Sun, May 21, 2017 at 3:05 AM, kant kodali <kanth...@gmail.com> wrote: >> >> > Looking at that ticket and reading the comments it looks like one of the >> > concern is as follows. >> > >> > "offsets.retention.minutes is designed to handle the case that a >> consumer >> > group goes away forever. In that case, we don't want to store the >> offsets >> > for that group forever." >> > >> > This can simply be addressed by setting offset.retention == >> log.retention >> > by default right? In which case offset wont be stored forever even when >> > consumer group goes away forever. When the consumer group goes away >> forever >> > the upper bound to clean up offsets would be equal to log.retention. >> > >> > >> > >> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <kanth...@gmail.com> >> wrote: >> > >> >> What is your average message size and network speed? >> >> >> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath < >> >> abhimanyunagr...@gmail.com> wrote: >> >> >> >>> Hi Girish, >> >>> >> >>> I did not set any value for offsets.retention.minutes so therefore >> what I >> >>> think is picking its default value i.e 1440 minutes so what do you >> think >> >>> what should I set if I am keeping my data for 30 days? >> >>> >> >>> Regards, >> >>> Abhimanyu >> >>> >> >> >> >> >> > >> > >
