Hi
One of our Kafka brokers was running out of disk space and when we checked
the file size in the kafka log dir we observed the following
$ du -h . --max-depth=2 | grep '__consumer_offsets'
4.0K ./kafka-logs/__consumer_offsets-16
4.0K ./kafka-logs/__consumer_offsets-40
35G ./kafka-logs/__consumer_offsets-44
4.0K ./kafka-logs/__consumer_offsets-8
4.0K ./kafka-logs/__consumer_offsets-38
4.0K ./kafka-logs/__consumer_offsets-20
4.0K ./kafka-logs/__consumer_offsets-34
4.0K ./kafka-logs/__consumer_offsets-18
4.0K ./kafka-logs/__consumer_offsets-32
251G ./kafka-logs/__consumer_offsets-14
4.0K ./kafka-logs/__consumer_offsets-4
4.0K ./kafka-logs/__consumer_offsets-26
4.0K ./kafka-logs/__consumer_offsets-12
4.0K ./kafka-logs/__consumer_offsets-30
4.0K ./kafka-logs/__consumer_offsets-6
4.0K ./kafka-logs/__consumer_offsets-2
4.0K ./kafka-logs/__consumer_offsets-24
4.0K ./kafka-logs/__consumer_offsets-36
4.0K ./kafka-logs/__consumer_offsets-46
4.0K ./kafka-logs/__consumer_offsets-42
4.0K ./kafka-logs/__consumer_offsets-22
4.0K ./kafka-logs/__consumer_offsets-0
4.0K ./kafka-logs/__consumer_offsets-28
4.0K ./kafka-logs/__consumer_offsets-10
4.0K ./kafka-logs/__consumer_offsets-48
As you can see, two of the log files (partition 44 and 14) have a huge
size. Do you have a hint to understand what could be happening here? May be
for some reason this partitions are not being compacted?
By the way, this is the description of the __consumer_offsets topic.
# ./bin/kafka-topics.sh --describe --zookeeper x.x.x.x:2181 --topic
__consumer_offsets
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:1
Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed
Topic: __consumer_offsets Partition: 0 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 1 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 2 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 3 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 4 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 5 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 6 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 7 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 8 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 9 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 10 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 11 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 12 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 13 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 14 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 15 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 16 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 17 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 18 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 19 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 20 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 21 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 22 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 23 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 24 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 25 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 26 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 27 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 28 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 29 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 30 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 31 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 32 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 33 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 34 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 35 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 36 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 37 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 38 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 39 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 40 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 41 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 42 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 43 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 44 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 45 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 46 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 47 Leader: 2
Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 48 Leader: 1
Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 49 Leader: 2
Replicas: 2 Isr: 2
Regards
Luciano
