Hi, Are you seeing any errors in log-cleaner.log? The log-cleaner thread can crash on certain errors.
Thanks Manikumar On Wed, Jul 13, 2016 at 9:54 PM, Lawrence Weikum <lwei...@pandora.com> wrote: > Hello, > > We’re seeing a strange behavior in Kafka 0.9.0.1 which occurs about every > other week. I’m curious if others have seen it and know of a solution. > > Setup and Scenario: > > - Brokers initially setup with log compaction turned off > > - After 30 days, log compaction was turned on > > - At this time, the number of Open FDs was ~ 30K per broker. > > - After 2 days, the __consumer_offsets topic was compacted > fully. Open FDs reduced to ~5K per broker. > > - Cluster has been under normal load for roughly 7 days. > > - At the 7 day mark, __consumer_offsets topic seems to have > stopped compacting on two of the brokers, and on those brokers, the FD > count is up to ~25K. > > > We have tried rebalancing the partitions before. The first time, the > destination broker had compacted the data fine and open FDs were low. The > second time, the destination broker kept the FDs open. > > > In all the broker logs, we’re seeing this messages: > INFO [Group Metadata Manager on Broker 8]: Removed 0 expired offsets in 0 > milliseconds. (kafka.coordinator.GroupMetadataManager) > > There are only 4 consumers at the moment on the cluster; one topic with 92 > partitions. > > Is there a reason why log compaction may stop working or why the > __consumer_offsets topic would start holding thousands of FDs? > > Thank you all for your help! > > Lawrence Weikum > >
