[
https://issues.apache.org/jira/browse/KAFKA-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Muralidhar Basani reassigned KAFKA-21377:
-----------------------------------------
Assignee: Muralidhar Basani
> kafka-topics --under-min-isr-partitions / --at-min-isr-partitions use the raw
> min.insync.replicas instead of the effective value used by the broker and
> controller
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-21377
> URL: https://issues.apache.org/jira/browse/KAFKA-21377
> Project: Kafka
> Issue Type: Bug
> Components: tools
> Affects Versions: 4.3.1, 4.4.0, 4.2.2
> Reporter: Rishi B
> Assignee: Muralidhar Basani
> Priority: Minor
>
> h3. Problem
> Since KIP-966, the broker and the controller use an effective min ISR of
> min(min.insync.replicas, replica count):
> * Broker: {{Partition#effectiveMinIsr}}. It drives the acks=all produce check
> and the UnderMinIsr/AtMinIsr metrics.
> * Controller: {{ReplicationControlManager#getTopicEffectiveMinIsr}}.
> KIP-966 states that this requirement "is not guarded by any feature
> flags/Metadata version", so it applies whether or not ELR is enabled.
> {{TopicCommand}} still compares the ISR size against the raw configured
> {{min.insync.replicas}}. For any topic whose {{min.insync.replicas}} is
> larger than its replication factor:
> * {{--under-min-isr-partitions}} lists partitions that the broker treats as
> healthy (at min ISR), and acks=all writes to them succeed.
> * {{--at-min-isr-partitions}} does not list them, although the broker counts
> them as at min ISR.
> Nothing prevents such a topic from existing:
> * Topic creation does not check min.insync.replicas against the replication
> factor.
> * A topic can also inherit a cluster or broker default that is higher than
> its replication factor.
> As a result, operators who follow the CLI open incidents for partitions that
> are not affected.
> h3. Steps to reproduce
> Environment: Apache Kafka 4.3.1 and 4.2.2, official {{apache/kafka}} Docker
> images, 3 KRaft nodes (combined broker+controller), PLAINTEXT. Newly
> formatted, so {{eligible.leader.replicas.version}} is finalized at 1.
> {code}
> bin/kafka-topics.sh --bootstrap-server kafka-1:19092 --create --topic
> minisr-gt-rf --partitions 1 --replication-factor 2 --config
> min.insync.replicas=3
> bin/kafka-topics.sh --bootstrap-server kafka-1:19092 --describe
> --under-min-isr-partitions
> bin/kafka-topics.sh --bootstrap-server kafka-1:19092 --describe
> --at-min-isr-partitions
> echo m1 | bin/kafka-console-producer.sh --bootstrap-server kafka-1:19092
> --topic minisr-gt-rf --command-property acks=all
> bin/kafka-get-offsets.sh --bootstrap-server kafka-1:19092 --topic
> minisr-gt-rf --time -1
> {code}
> *Actual* (4.3.1; 4.2.2 is identical apart from the leader id)
> * {{--under-min-isr-partitions}} prints {{Topic: minisr-gt-rf Partition: 0
> Leader: 3 Replicas: 3,1 Isr: 3,1 Elr: LastKnownElr:}}
> * {{--at-min-isr-partitions}} prints nothing.
> * The acks=all produce succeeds (exit code 0), and {{kafka-get-offsets}}
> returns {{minisr-gt-rf:0:1}}.
> * On the leader broker:
> ** {{kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount}} stays
> 0.
> ** {{kafka.server:type=ReplicaManager,name=AtMinIsrPartitionCount}} goes from
> 0 to 1.
> **
> {{kafka.cluster:type=Partition,name=UnderMinIsr,topic=minisr-gt-rf,partition=0}}
> = 0 and {{...name=AtMinIsr,...}} = 1.
> *Same result with ELR disabled*
> {code}
> bin/kafka-features.sh --bootstrap-server kafka-1:19092 downgrade --feature
> eligible.leader.replicas.version=0
> bin/kafka-topics.sh --bootstrap-server kafka-1:19092 --create --topic
> rf1-minisr2 --partitions 1 --replication-factor 1 --config
> min.insync.replicas=2
> {code}
> * {{--under-min-isr-partitions}} lists {{rf1-minisr2}} partition 0
> ({{Replicas: 1 Isr: 1}}), and {{--at-min-isr-partitions}} does not.
> * The acks=all produce succeeds, and the offset advances to 1.
> * On the leader, {{UnderMinIsr}} = 0, {{AtMinIsr}} = 1, and
> {{UnderMinIsrPartitionCount}} stays 0.
> *Expected*
> The CLI agrees with the broker: the partition is not listed by
> {{--under-min-isr-partitions}} and is listed by {{--at-min-isr-partitions}}.
> h3. Cause
> {{tools/src/main/java/org/apache/kafka/tools/TopicCommand.java}}
> ({{PartitionDescription}}, 4.3 branch around L314-327):
> {code:java}
> int minIsrCount() {
> return
> Integer.parseInt(config.get(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG).value());
> }
> boolean isUnderMinIsr() {
> return !hasLeader() || info.isr().size() < minIsrCount();
> }
> boolean isAtMinIsrPartitions() {
> return minIsrCount() == info.isr().size();
> }
> {code}
> The option help (around L778-779) also says "isr count is less than the
> configured minimum".
> The same code and help text are unchanged in the 4.4 branch and on trunk
> (L314-332, L779-781).
> {{core/src/main/scala/kafka/cluster/Partition.scala}} (4.3 branch around
> L246-248; the same code is at L391-393 in 4.1 and identical in 3.9/4.0):
> {code:scala}
> private def effectiveMinIsr(leaderLog: UnifiedLog): Int = {
> leaderLog.config.minInSyncReplicas.min(remoteReplicasMap.size + 1)
> }
> {code}
> It is used by {{isUnderMinIsr}}, {{isAtMinIsr}}, the acks=all completion
> check and {{appendRecordsToLeader}}. None of these depend on ELR.
> {{metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java}}
> ({{getTopicEffectiveMinIsr}}, 4.3 branch around L2463-2468) applies the same
> cap on the controller with {{Math.min(currentMinIsr, replicationFactor)}}.
> h3. Proposed fix
> * Compute the same effective value in {{TopicCommand}}, e.g.
> {{Math.min(configured, info.replicas().size())}}, and use it in
> {{isUnderMinIsr()}} and {{isAtMinIsrPartitions()}}.
> * Update the help text of both options ("less than / equal to the configured
> minimum") to say the effective minimum (min of {{min.insync.replicas}} and
> the replication factor).
> * Optionally, show the effective value next to the configured one in
> {{--describe}} output when they differ.
> * Add tests in {{TopicCommandTest}} for a topic with {{min.insync.replicas}}
> > replication factor.
> h3. Related
> * KIP-966 (Eligible Leader Replicas): introduced the effective min ISR. The
> KIP notes that it is not gated by a feature flag or metadata version.
> * KIP-1372 / KAFKA-19726 (under discussion): would reject new
> {{min.insync.replicas}} > replication factor configs. The CLI fix is still
> needed because:
> ** Existing topics keep the mismatch.
> ** Topics can fall back to a cluster or broker default above their
> replication factor (KIP-1372 keeps DELETE allowed).
> ** Topic creation itself does not validate min.insync.replicas against the
> replication factor today.
> * KIP-351 / KAFKA-7236: added {{--under-min-isr-partitions}} before the
> broker-side cap existed.
> * KAFKA-4680 (open): {{min.insync.replicas}} can be set higher than the
> replication factor. It dates from when such writes were rejected. Since the
> effective-min-ISR cap, they are accepted.
> * KAFKA-8695 (resolved as a duplicate of KAFKA-4680): UnderReplicated and
> UnderMinIsr metrics diverge when the configuration is inconsistent.
> * KAFKA-20235 (open): expose EffectiveMinIsr as a partition metric.
> Complementary to this issue, since operators could then compare the CLI
> against the effective value.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
