Hello, I posted this on StackOverflow
<https://stackoverflow.com/questions/47948399/kafka-per-topic-retention-bytes-and-global-log-retention-bytes-not-working>also
but haven't gotten any response.
thanks in advance,
Thunder
__________
We are running a 6 node cluster of kafka 0.11.0. We have set a global as
well as a per-topic retention in bytes, neither of which is being applied.
There are no errors that I can see in the logs, just nothing being deleted
(by size; the time retention does seem to be working)
See relevant configs below:
*./config/server.properties* :
# global retention 75GB or 60 days, segment size 512MB
log.retention.bytes=75000000000
log.retention.check.interval.ms=60000
log.retention.hours=1440
log.cleanup.policy=delete
log.segment.bytes=536870912
*topic configuration (30GB):*
[tstumpges@kafka-02 kafka]$ bin/kafka-topics.sh --zookeeper
zk-01:2181/kafka --describe --topic stg_logtopic
Topic:stg_logtopic PartitionCount:12 ReplicationFactor:3
Configs:retention.bytes=30000000000
Topic: stg_logtopic Partition: 0 Leader: 4
Replicas: 4,5,6 Isr: 4,5,6
Topic: stg_logtopic Partition: 1 Leader: 5
Replicas: 5,6,1 Isr: 5,1,6
...
And, disk usage showing 910GB usage for one partition!
[tstumpges@kafka-02 kafka]$ sudo du -s -h /data1/kafka-data/*
82G /data1/kafka-data/stg_logother3-2
155G /data1/kafka-data/stg_logother2-9
169G /data1/kafka-data/stg_logother1-6
910G /data1/kafka-data/stg_logtopic-4
I can see there are plenty of segment log files (512MB each) in the
partition directory... what is going on?!
Thanks in advance, Thunder
