zhifeng.peng created KAFKA-10268:
------------------------------------
Summary: dynamic config like "--delete-config log.retention.ms"
not work
Key: KAFKA-10268
URL: https://issues.apache.org/jira/browse/KAFKA-10268
Project: Kafka
Issue Type: Bug
Components: log, log cleaner
Affects Versions: 2.1.1
Reporter: zhifeng.peng
Attachments: server.log.2020-07-13-14
After I set "log.retention.ms=301000" to clean the data,i use the cmd
"bin/kafka-configs.sh --bootstrap-server 10.129.104.15:9092 --entity-type
brokers --entity-default --alter --delete-config log.retention.ms" to reset to
default. Static broker configuration like log.retention.hours is 168h and no
topic level configuration like retention.ms.
But it did not take effect actually although server.log print the broker
configuration like that.
log.retention.check.interval.ms = 300000
log.retention.hours = 168
log.retention.minutes = null
{color:#FF0000}log.retention.ms = null{color}
log.roll.hours = 168
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 1073741824
log.segment.delete.delay.ms = 60000
Then we can see that retention time is still 301000ms from the server.log and
segments have been deleted.
[2020-07-13 14:30:00,958] INFO [Log partition=test_retention-2,
dir=/data/kafka_logs-test] Found deletable segments with base offsets
[5005329,6040360] due to retention time 301000ms breach (kafka.log.Log)
[2020-07-13 14:30:00,959] INFO [Log partition=test_retention-2,
dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 5005329, size
1073741222] for deletion. (kafka.log.Log)
[2020-07-13 14:30:00,959] INFO [Log partition=test_retention-2,
dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 6040360, size
1073728116] for deletion. (kafka.log.Log)
[2020-07-13 14:30:00,959] INFO [Log partition=test_retention-2,
dir=/data/kafka_logs-test] Incrementing log start offset to 7075648
(kafka.log.Log)
[2020-07-13 14:30:00,960] INFO [Log partition=test_retention-0,
dir=/data/kafka_logs-test] Found deletable segments with base offsets
[5005330,6040410] due to retention time 301000ms breach (kafka.log.Log)
[2020-07-13 14:30:00,960] INFO [Log partition=test_retention-0,
dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 5005330, size
1073732368] for deletion. (kafka.log.Log)
[2020-07-13 14:30:00,961] INFO [Log partition=test_retention-0,
dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 6040410, size
1073735366] for deletion. (kafka.log.Log)
[2020-07-13 14:30:00,961] INFO [Log partition=test_retention-0,
dir=/data/kafka_logs-test] Incrementing log start offset to 7075685
(kafka.log.Log)
[2020-07-13 14:31:00,959] INFO [Log partition=test_retention-2,
dir=/data/kafka_logs-test] Deleting segment 5005329 (kafka.log.Log)
[2020-07-13 14:31:00,959] INFO [Log partition=test_retention-2,
dir=/data/kafka_logs-test] Deleting segment 6040360 (kafka.log.Log)
[2020-07-13 14:31:00,961] INFO [Log partition=test_retention-0,
dir=/data/kafka_logs-test] Deleting segment 5005330 (kafka.log.Log)
[2020-07-13 14:31:00,961] INFO [Log partition=test_retention-0,
dir=/data/kafka_logs-test] Deleting segment 6040410 (kafka.log.Log)
[2020-07-13 14:31:01,144] INFO Deleted log
/data/kafka_logs-test/test_retention-2/00000000000006040360.log.deleted.
(kafka.log.LogSegment)
[2020-07-13 14:31:01,144] INFO Deleted offset index
/data/kafka_logs-test/test_retention-2/00000000000006040360.index.deleted.
(kafka.log.LogSegment)
[2020-07-13 14:31:01,144] INFO Deleted time index
/data/kafka_logs-test/test_retention-2/00000000000006040360.timeindex.deleted.
(kafka.log.LogSegment)
Here are a few steps to reproduce it.
1、set log.retention.ms=301000:
bin/kafka-configs.sh --bootstrap-server 10.129.104.15:9092 --entity-type
brokers --entity-default --alter --add-config log.retention.ms=301000
2、produce messages to the topic:
bin/kafka-producer-perf-test.sh --topic test_retention --num-records 10000000
--throughput -1 --producer-props bootstrap.servers=10.129.104.15:9092
--record-size 1024
3、reset log.retention.ms to the default:
bin/kafka-configs.sh --bootstrap-server 10.129.104.15:9092 --entity-type
brokers --entity-default --alter --delete-config log.retention.ms
I have attched server.log. You can see the log from row 238 to row 731.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
