[
https://issues.apache.org/jira/browse/KAFKA-12946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17363584#comment-17363584
]
Ron Dagostino commented on KAFKA-12946:
---------------------------------------
Yeah, there are bugs. The KIP I referred to mentions one. There have also
been several changes to make the log cleaner thread more robust to failure over
time — even since the 2.0 version you are on. Upgrading might not help
immediately, but you will want to leverage the KIP-664 tools at some point, so
best to keep current. You should definitely read that KIP.
> __consumer_offsets topic with very big partitions
> -------------------------------------------------
>
> Key: KAFKA-12946
> URL: https://issues.apache.org/jira/browse/KAFKA-12946
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
> Affects Versions: 2.0.0
> Reporter: Emi
> Priority: Critical
>
> I am using Kafka 2.0.0 with java 8u191
> There is a partitions of the __consumer_offsets topic that is 600 GB with
> 6000 segments older than 4 months. Other partitions of that topic are small:
> 20-30 MB.
> There are 60 consumer groups, 90 topics and 100 partitions per topic.
> There aren't errors in the logs. From the log of the logcleaner, I can see
> that partition is never touched from the logcleaner thread for the
> compaction, but it only add new segments.
> How is this possible?
> There was another partition with the same problem, but after some months it
> has been compacted. Now there is only one partition with this problem, but
> this is bigger and keep growing
> I have used the kafka-dump-log tool to check these old segments and I can see
> many duplicates. So I would assume that is not compacted.
> My settings:
> {{offsets.commit.required.acks = -1}}
> {{[offsets.commit.timeout.ms|http://offsets.commit.timeout.ms/]}} = 5000
> {{offsets.load.buffer.size = 5242880}}
>
> {{[offsets.retention.check.interval.ms|http://offsets.retention.check.interval.ms/]}}
> = 600000
> {{offsets.retention.minutes = 10080}}
> {{offsets.topic.compression.codec = 0}}
> {{offsets.topic.num.partitions = 50}}
> {{offsets.topic.replication.factor = 3}}
> {{offsets.topic.segment.bytes = 104857600}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
