Rishi B created KAFKA-21377:
-------------------------------
Summary: kafka-topics --under-min-isr-partitions /
--at-min-isr-partitions use the raw min.insync.replicas instead of the
effective value used by the broker and controller
Key: KAFKA-21377
URL: https://issues.apache.org/jira/browse/KAFKA-21377
Project: Kafka
Issue Type: Bug
Components: tools
Affects Versions: 4.3.1, 4.2.2
Reporter: Rishi B
h3. Problem
Since KIP-966, the broker and the controller use an effective min ISR of
min(min.insync.replicas, replica count):
* Broker: {{Partition#effectiveMinIsr}}. It drives the acks=all produce check
and the UnderMinIsr/AtMinIsr metrics.
* Controller: {{ReplicationControlManager#getTopicEffectiveMinIsr}}.
KIP-966 states that this requirement "is not guarded by any feature
flags/Metadata version", so it applies whether or not ELR is enabled.
{{TopicCommand}} still compares the ISR size against the raw configured
{{min.insync.replicas}}. For any topic whose {{min.insync.replicas}} is larger
than its replication factor:
* {{--under-min-isr-partitions}} lists partitions that the broker treats as
healthy (at min ISR), and acks=all writes to them succeed.
* {{--at-min-isr-partitions}} does not list them, although the broker counts
them as at min ISR.
Nothing prevents such a topic from existing:
* Topic creation does not check min.insync.replicas against the replication
factor.
* A topic can also inherit a cluster or broker default that is higher than its
replication factor.
As a result, operators who follow the CLI open incidents for partitions that
are not affected.
h3. Steps to reproduce
Environment: Apache Kafka 4.3.1 and 4.2.2, official {{apache/kafka}} Docker
images, 3 KRaft nodes (combined broker+controller), PLAINTEXT. Newly formatted,
so {{eligible.leader.replicas.version}} is finalized at 1.
{code}
bin/kafka-topics.sh --bootstrap-server kafka-1:19092 --create --topic
minisr-gt-rf --partitions 1 --replication-factor 2 --config
min.insync.replicas=3
bin/kafka-topics.sh --bootstrap-server kafka-1:19092 --describe
--under-min-isr-partitions
bin/kafka-topics.sh --bootstrap-server kafka-1:19092 --describe
--at-min-isr-partitions
echo m1 | bin/kafka-console-producer.sh --bootstrap-server kafka-1:19092
--topic minisr-gt-rf --command-property acks=all
bin/kafka-get-offsets.sh --bootstrap-server kafka-1:19092 --topic minisr-gt-rf
--time -1
{code}
*Actual* (4.3.1; 4.2.2 is identical apart from the leader id)
* {{--under-min-isr-partitions}} prints {{Topic: minisr-gt-rf Partition: 0
Leader: 3 Replicas: 3,1 Isr: 3,1 Elr: LastKnownElr:}}
* {{--at-min-isr-partitions}} prints nothing.
* The acks=all produce succeeds (exit code 0), and {{kafka-get-offsets}}
returns {{minisr-gt-rf:0:1}}.
* On the leader broker:
** {{kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount}} stays 0.
** {{kafka.server:type=ReplicaManager,name=AtMinIsrPartitionCount}} goes from 0
to 1.
**
{{kafka.cluster:type=Partition,name=UnderMinIsr,topic=minisr-gt-rf,partition=0}}
= 0 and {{...name=AtMinIsr,...}} = 1.
*Same result with ELR disabled*
{code}
bin/kafka-features.sh --bootstrap-server kafka-1:19092 downgrade --feature
eligible.leader.replicas.version=0
bin/kafka-topics.sh --bootstrap-server kafka-1:19092 --create --topic
rf1-minisr2 --partitions 1 --replication-factor 1 --config min.insync.replicas=2
{code}
* {{--under-min-isr-partitions}} lists {{rf1-minisr2}} partition 0 ({{Replicas:
1 Isr: 1}}), and {{--at-min-isr-partitions}} does not.
* The acks=all produce succeeds, and the offset advances to 1.
* On the leader, {{UnderMinIsr}} = 0, {{AtMinIsr}} = 1, and
{{UnderMinIsrPartitionCount}} stays 0.
*Expected*
The CLI agrees with the broker: the partition is not listed by
{{--under-min-isr-partitions}} and is listed by {{--at-min-isr-partitions}}.
h3. Cause
{{tools/src/main/java/org/apache/kafka/tools/TopicCommand.java}}
({{PartitionDescription}}, 4.3 branch around L314-327):
{code:java}
int minIsrCount() {
return
Integer.parseInt(config.get(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG).value());
}
boolean isUnderMinIsr() {
return !hasLeader() || info.isr().size() < minIsrCount();
}
boolean isAtMinIsrPartitions() {
return minIsrCount() == info.isr().size();
}
{code}
The option help (around L778-779) also says "isr count is less than the
configured minimum".
The same code and help text are unchanged in the 4.4 branch and on trunk
(L314-332, L779-781).
{{core/src/main/scala/kafka/cluster/Partition.scala}} (4.3 branch around
L246-248; the same code is at L391-393 in 4.1 and identical in 3.9/4.0):
{code:scala}
private def effectiveMinIsr(leaderLog: UnifiedLog): Int = {
leaderLog.config.minInSyncReplicas.min(remoteReplicasMap.size + 1)
}
{code}
It is used by {{isUnderMinIsr}}, {{isAtMinIsr}}, the acks=all completion check
and {{appendRecordsToLeader}}. None of these depend on ELR.
{{metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java}}
({{getTopicEffectiveMinIsr}}, 4.3 branch around L2463-2468) applies the same
cap on the controller with {{Math.min(currentMinIsr, replicationFactor)}}.
h3. Proposed fix
* Compute the same effective value in {{TopicCommand}}, e.g.
{{Math.min(configured, info.replicas().size())}}, and use it in
{{isUnderMinIsr()}} and {{isAtMinIsrPartitions()}}.
* Update the help text of both options ("less than / equal to the configured
minimum") to say the effective minimum (min of {{min.insync.replicas}} and the
replication factor).
* Optionally, show the effective value next to the configured one in
{{--describe}} output when they differ.
* Add tests in {{TopicCommandTest}} for a topic with {{min.insync.replicas}} >
replication factor.
h3. Related
* KIP-966 (Eligible Leader Replicas): introduced the effective min ISR. The KIP
notes that it is not gated by a feature flag or metadata version.
* KIP-1372 / KAFKA-19726 (under discussion): would reject new
{{min.insync.replicas}} > replication factor configs. The CLI fix is still
needed because:
** Existing topics keep the mismatch.
** Topics can fall back to a cluster or broker default above their replication
factor (KIP-1372 keeps DELETE allowed).
** Topic creation itself does not validate min.insync.replicas against the
replication factor today.
* KIP-351 / KAFKA-7236: added {{--under-min-isr-partitions}} before the
broker-side cap existed.
* KAFKA-4680 (open): {{min.insync.replicas}} can be set higher than the
replication factor. It dates from when such writes were rejected. Since the
effective-min-ISR cap, they are accepted.
* KAFKA-8695 (resolved as a duplicate of KAFKA-4680): UnderReplicated and
UnderMinIsr metrics diverge when the configuration is inconsistent.
* KAFKA-20235 (open): expose EffectiveMinIsr as a partition metric.
Complementary to this issue, since operators could then compare the CLI against
the effective value.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
