[
https://issues.apache.org/jira/browse/KAFKA-13473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Patterson updated KAFKA-13473:
----------------------------------
Description:
Upon restarting kafka, dynamically configured log cleaner configs aren't picked
up and applied.
Here are some logs from a local kafka when I up the threads to 2 using the
kafka-config tool - Noting the last 2 lines where it starts up 2 log cleaner
threads.
{code:java}
[2021-11-23 21:09:50,044] INFO [Admin Manager on Broker 1001]: Updating brokers
with new configuration : log.cleaner.threads -> 2 (kafka.server.ZkAdminManager)
[2021-11-23 21:09:50,092] INFO Processing override for entityPath:
brokers/<default> with config: HashMap(log.cleaner.threads -> 2)
(kafka.server.DynamicConfigManager) log.cleaner.threads = 2
[2021-11-23 21:09:50,113] INFO Shutting down the log cleaner.
(kafka.log.LogCleaner)
[2021-11-23 21:09:50,114] INFO [kafka-log-cleaner-thread-0]: Shutting down
(kafka.log.LogCleaner)
[2021-11-23 21:09:50,116] INFO [kafka-log-cleaner-thread-0]: Stopped
(kafka.log.LogCleaner)
[2021-11-23 21:09:50,116] INFO [kafka-log-cleaner-thread-0]: Shutdown completed
(kafka.log.LogCleaner)
[2021-11-23 21:09:50,119] INFO Starting the log cleaner (kafka.log.LogCleaner)
[2021-11-23 21:09:50,178] INFO [kafka-log-cleaner-thread-0]: Starting
(kafka.log.LogCleaner)
[2021-11-23 21:09:50,181] INFO [kafka-log-cleaner-thread-1]: Starting
(kafka.log.LogCleaner){code}
And now after a restart, at no point does it ever start 2 threads, even though
it clearly knows about the configs
{code:java}
[2021-11-23 21:10:46,659] INFO Starting the log cleaner (kafka.log.LogCleaner)
[2021-11-23 21:10:46,723] INFO [kafka-log-cleaner-thread-0]: Starting
(kafka.log.LogCleaner)
[2021-11-23 21:10:48,124] INFO Processing override for entityPath:
brokers/<default> with config: HashMap(log.cleaner.threads -> 2)
(kafka.server.DynamicConfigManager) log.cleaner.backoff.ms = 15000
log.cleaner.dedupe.buffer.size = 150000000 log.cleaner.delete.retention.ms =
86400000 log.cleaner.enable = true log.cleaner.io.buffer.load.factor = 0.9
log.cleaner.io.buffer.size = 524288 log.cleaner.io.max.bytes.per.second =
1.7976931348623157E308 log.cleaner.max.compaction.lag.ms = 9223372036854775807
log.cleaner.min.cleanable.ratio = 0.5 log.cleaner.min.compaction.lag.ms = 0
log.cleaner.threads = 2{code}
When investigating from the kafka config tool all looks well.
{code:java}
kafka-configs --bootstrap-server $BROKER_URL --entity-type brokers
--entity-default --describe --all | grep log.cleaner.threads
log.cleaner.threads=2 sensitive=false
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.threads=2}{code}
But if you try change the config you soon find out that all is not well(note
here it mentions the current value is 1 in the validation message)
{code:java}
kafka-configs --bootstrap-server $BROKER_URL --entity-type brokers
--entity-default --alter --add-config log.cleaner.threads=3 Error while
executing config command with args '--bootstrap-server profile_kafka:9093
--entity-type brokers --entity-default --alter --add-config
log.cleaner.threads=3' java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.InvalidRequestException: Invalid config value
for resource ConfigResource(type=BROKER, name=''): Invalid value
org.apache.kafka.common.config.ConfigException: Log cleaner threads cannot be
increased to more than double the current value 1 for configuration Invalid
dynamic configuration{code}
was:
Upon restarting kafka, dynamically configured log cleaner configs aren't picked
up and applied.
Here are some logs from a local kafka when I up the threads to 2 using the
kafka-config tool - Noting the last 2 lines where it starts up 2 log cleaner
threads.
{code:java}
[2021-11-23 21:09:50,044] INFO [Admin Manager on Broker 1001]: Updating brokers
with new configuration : log.cleaner.threads -> 2 (kafka.server.ZkAdminManager)
[2021-11-23 21:09:50,092] INFO Processing override for entityPath:
brokers/<default> with config: HashMap(log.cleaner.threads -> 2)
(kafka.server.DynamicConfigManager) log.cleaner.threads = 2 [2021-11-23
21:09:50,113] INFO Shutting down the log cleaner. (kafka.log.LogCleaner)
[2021-11-23 21:09:50,114] INFO [kafka-log-cleaner-thread-0]: Shutting down
(kafka.log.LogCleaner) [2021-11-23 21:09:50,116] INFO
[kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner) [2021-11-23
21:09:50,116] INFO [kafka-log-cleaner-thread-0]: Shutdown completed
(kafka.log.LogCleaner) [2021-11-23 21:09:50,119] INFO Starting the log cleaner
(kafka.log.LogCleaner) [2021-11-23 21:09:50,178] INFO
[kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner) [2021-11-23
21:09:50,181] INFO [kafka-log-cleaner-thread-1]: Starting
(kafka.log.LogCleaner){code}
And now after a restart, at no point does it ever start 2 threads, even though
it clearly knows about the configs
{code:java}
[2021-11-23 21:10:46,659] INFO Starting the log cleaner (kafka.log.LogCleaner)
[2021-11-23 21:10:46,723] INFO [kafka-log-cleaner-thread-0]: Starting
(kafka.log.LogCleaner) [2021-11-23 21:10:48,124] INFO Processing override for
entityPath: brokers/<default> with config: HashMap(log.cleaner.threads -> 2)
(kafka.server.DynamicConfigManager) log.cleaner.backoff.ms = 15000
log.cleaner.dedupe.buffer.size = 150000000 log.cleaner.delete.retention.ms =
86400000 log.cleaner.enable = true log.cleaner.io.buffer.load.factor = 0.9
log.cleaner.io.buffer.size = 524288 log.cleaner.io.max.bytes.per.second =
1.7976931348623157E308 log.cleaner.max.compaction.lag.ms = 9223372036854775807
log.cleaner.min.cleanable.ratio = 0.5 log.cleaner.min.compaction.lag.ms = 0
log.cleaner.threads = 2{code}
When investigating from the kafka config tool all looks well.
{code:java}
kafka-configs --bootstrap-server $BROKER_URL --entity-type brokers
--entity-default --describe --all | grep log.cleaner.threads
log.cleaner.threads=2 sensitive=false
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.threads=2}{code}
But if you try change the config you soon find out that all is not well(note
here it mentions the current value is 1 in the validation message)
{code:java}
kafka-configs --bootstrap-server $BROKER_URL --entity-type brokers
--entity-default --alter --add-config log.cleaner.threads=3 Error while
executing config command with args '--bootstrap-server profile_kafka:9093
--entity-type brokers --entity-default --alter --add-config
log.cleaner.threads=3' java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.InvalidRequestException: Invalid config value
for resource ConfigResource(type=BROKER, name=''): Invalid value
org.apache.kafka.common.config.ConfigException: Log cleaner threads cannot be
increased to more than double the current value 1 for configuration Invalid
dynamic configuration{code}
> Log cleaner Dynamic configs aren't applied after a restart
> ----------------------------------------------------------
>
> Key: KAFKA-13473
> URL: https://issues.apache.org/jira/browse/KAFKA-13473
> Project: Kafka
> Issue Type: Bug
> Components: config, core
> Affects Versions: 2.8.1
> Reporter: Tim Patterson
> Priority: Minor
>
> Upon restarting kafka, dynamically configured log cleaner configs aren't
> picked up and applied.
>
> Here are some logs from a local kafka when I up the threads to 2 using the
> kafka-config tool - Noting the last 2 lines where it starts up 2 log cleaner
> threads.
>
> {code:java}
> [2021-11-23 21:09:50,044] INFO [Admin Manager on Broker 1001]: Updating
> brokers with new configuration : log.cleaner.threads -> 2
> (kafka.server.ZkAdminManager)
> [2021-11-23 21:09:50,092] INFO Processing override for entityPath:
> brokers/<default> with config: HashMap(log.cleaner.threads -> 2)
> (kafka.server.DynamicConfigManager) log.cleaner.threads = 2
> [2021-11-23 21:09:50,113] INFO Shutting down the log cleaner.
> (kafka.log.LogCleaner)
> [2021-11-23 21:09:50,114] INFO [kafka-log-cleaner-thread-0]: Shutting down
> (kafka.log.LogCleaner)
> [2021-11-23 21:09:50,116] INFO [kafka-log-cleaner-thread-0]: Stopped
> (kafka.log.LogCleaner)
> [2021-11-23 21:09:50,116] INFO [kafka-log-cleaner-thread-0]: Shutdown
> completed (kafka.log.LogCleaner)
> [2021-11-23 21:09:50,119] INFO Starting the log cleaner (kafka.log.LogCleaner)
> [2021-11-23 21:09:50,178] INFO [kafka-log-cleaner-thread-0]: Starting
> (kafka.log.LogCleaner)
> [2021-11-23 21:09:50,181] INFO [kafka-log-cleaner-thread-1]: Starting
> (kafka.log.LogCleaner){code}
> And now after a restart, at no point does it ever start 2 threads, even
> though it clearly knows about the configs
>
> {code:java}
> [2021-11-23 21:10:46,659] INFO Starting the log cleaner (kafka.log.LogCleaner)
> [2021-11-23 21:10:46,723] INFO [kafka-log-cleaner-thread-0]: Starting
> (kafka.log.LogCleaner)
> [2021-11-23 21:10:48,124] INFO Processing override for entityPath:
> brokers/<default> with config: HashMap(log.cleaner.threads -> 2)
> (kafka.server.DynamicConfigManager) log.cleaner.backoff.ms = 15000
> log.cleaner.dedupe.buffer.size = 150000000 log.cleaner.delete.retention.ms =
> 86400000 log.cleaner.enable = true log.cleaner.io.buffer.load.factor = 0.9
> log.cleaner.io.buffer.size = 524288 log.cleaner.io.max.bytes.per.second =
> 1.7976931348623157E308 log.cleaner.max.compaction.lag.ms =
> 9223372036854775807 log.cleaner.min.cleanable.ratio = 0.5
> log.cleaner.min.compaction.lag.ms = 0 log.cleaner.threads = 2{code}
>
> When investigating from the kafka config tool all looks well.
>
> {code:java}
> kafka-configs --bootstrap-server $BROKER_URL --entity-type brokers
> --entity-default --describe --all | grep log.cleaner.threads
> log.cleaner.threads=2 sensitive=false
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.threads=2}{code}
>
> But if you try change the config you soon find out that all is not well(note
> here it mentions the current value is 1 in the validation message)
>
> {code:java}
> kafka-configs --bootstrap-server $BROKER_URL --entity-type brokers
> --entity-default --alter --add-config log.cleaner.threads=3 Error while
> executing config command with args '--bootstrap-server profile_kafka:9093
> --entity-type brokers --entity-default --alter --add-config
> log.cleaner.threads=3' java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.InvalidRequestException: Invalid config value
> for resource ConfigResource(type=BROKER, name=''): Invalid value
> org.apache.kafka.common.config.ConfigException: Log cleaner threads cannot be
> increased to more than double the current value 1 for configuration Invalid
> dynamic configuration{code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
