Tamas Kornai created KAFKA-20890:
------------------------------------
Summary: KRaft broker startup can fail to restore persisted
dynamic broker configs when the local metadata snapshot is missing or stale
Key: KAFKA-20890
URL: https://issues.apache.org/jira/browse/KAFKA-20890
Project: Kafka
Issue Type: Bug
Components: config, core, kraft
Affects Versions: 4.3.0, 4.2.0, 4.1.0, 4.0.0, 3.9.2
Reporter: Tamas Kornai
On KRaft broker startup, persisted dynamic broker configurations may not be
available when runtime components and their Reconfigurable implementations are
initialized. When the latest local metadata snapshot is missing or does not yet
contain the relevant ConfigRecord, the broker starts from its static
configuration and receives the persisted dynamic configuration later through
metadata-log replay.
That replay is processed as a live reconfiguration. A configuration whose final
persisted value is valid can therefore be rejected by validation that compares
the requested value with the component's current runtime state. The broker
remains on its static value even though the dynamic configuration store, and
consequently DescribeConfigs, can report the persisted dynamic value.
This was originally observed with num.recovery.threads.per.data.dir, but the
underlying problem is not specific to recovery-thread configuration. It can
affect dynamic configurations whose validation depends on current runtime
state, initialization order, or another dynamically supplied setting.
## Concrete production observation
The static broker value was:
num.recovery.threads.per.data.dir=1
A cluster-default dynamic configuration was rolled out in two safe steps:
unset -> 2 -> 4
The original rollout succeeded. Brokers logged both runtime transitions:
Resizing recovery thread pool size for each data dir from 1 to 2
Resizing recovery thread pool size for each data dir from 2 to 4
During a subsequent ZooKeeper-to-KRaft transition, affected brokers rebuilt
their configuration as:
1 -> 4 -> 1
Their final runtime KafkaConfig value was 1, while the stored cluster-default
dynamic configuration continued to report 4 and there were no per-broker
overrides.
Later attempts to apply 4 were rejected because the active runtime value was
still 1:
Dynamic thread count update validation failed for
num.recovery.threads.per.data.dir=4,
value should not be greater than double the current value 1
## Expected behavior
* A broker restart restores the final persisted dynamic configuration before
dependent runtime components become active, regardless of whether the latest
local metadata snapshot happens to contain that configuration.
* - Startup restoration is not rejected merely because the persisted final
value is more than a live-update limit away from the static value.
* - If applying a dynamic configuration fails, configuration APIs do not
report that rejected value as the broker's effective configuration.
## Actual behavior
* Without a suitable local snapshot, the broker initializes affected
components using the static value.
* - Metadata-log replay later presents the persisted value as a live update.
* - Current-state-dependent validation can reject that update.
* - The component remains on the static runtime value.
* - The dynamic configuration cache can nevertheless contain and report the
rejected persisted value, so the reported effective configuration differs from
the active runtime state.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
