[
https://issues.apache.org/jira/browse/KAFKA-21419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haifeng Chen updated KAFKA-21419:
---------------------------------
Description:
When a partition reassignment is still in progress and a new reassignment for
the same partition is submitted (for example with
{{{}kafka-reassign-partitions.sh --execute --additional{}}}, which Cruise
Control does routinely), the KRaft controller computes the new
{{addingReplicas}} as {{{}target − currentReplicas{}}}. {{currentReplicas}}
already contains the replicas the first reassignment is still adding, so any of
them that are also in the new target are silently promoted to plain replicas
even though they have not joined the ISR yet.
The override then completes as soon as the replicas it newly adds are in sync,
leaving the partition with a replica that is out of the ISR and that no
reassignment accounts for: {{{}-{}}}list shows nothing in flight, {{-verify}}
reports success, under-replicated partitions stay non-zero until the replica
happens to catch up, and the removing replica is dropped while the partition
has fewer in-sync replicas than it looks.
Reproduction (any KRaft cluster):
# Partition on {{{}[1,2,3]{}}}; reassign to {{[1,2,4]}} and hold broker 4 out
of the ISR (e.g. 1 B/s follower throttle). {{--list}} shows {{adding: 4.
removing: 3.}}
# While 4 is still adding, run {{--execute --additional}} with
{{{}[1,2,4,5]{}}}.
# {{--list}} shows {{adding: 5. removing: 3.}} — 4 is gone from adding.
# Once 5 is in sync the reassignment completes: {{{}replicas=[1,2,4,5]
isr=[1,2,5]{}}}, no reassignment in progress.
Expected: step 3 shows {{adding: 4,5. removing: 3.}} and the reassignment stays
in progress until 4 is also in sync.
The override case is only correct today when the new adding set happens to be
empty (the builder leaves the old one in place) or when the RF-reduction guard
{{adding.size() < removing.size()}} blocks completion, which masks the bug for
shrinking overrides.
ZooKeeper-mode controllers computed the new adding set relative to the original
replica set, so this is a KRaft regression.
Fix: derive the new adding set as {{{}target − (currentReplicas −
currentAddingReplicas){}}}, so a replica that was still being added by the
previous reassignment stays in {{addingReplicas}} if the new target keeps it.
was:
When a partition reassignment is still in progress and a new reassignment for
the same partition is submitted (for example with
{{{}kafka-reassign-partitions.sh --execute --additional{}}}, which Cruise
Control does routinely), the KRaft controller computes the new
{{addingReplicas}} as {{{}target − currentReplicas{}}}. {{currentReplicas}}
already contains the replicas the first reassignment is still adding, so any of
them that are also in the new target are silently promoted to plain replicas
even though they have not joined the ISR yet.
The override then completes as soon as the replicas it newly adds are in sync,
leaving the partition with a replica that is out of the ISR and that no
reassignment accounts for: {{--list}} shows nothing in flight, {{--verify}}
reports success, under-replicated partitions stay non-zero until the replica
happens to catch up, and the removing replica is dropped while the partition
has fewer in-sync replicas than it looks.
Reproduction (any KRaft cluster):
# Partition on {{{}[1,2,3]{}}}; reassign to {{[1,2,4]}} and hold broker 4 out
of the ISR (e.g. 1 B/s follower throttle). {{--list}} shows {{adding: 4.
removing: 3.}}
# While 4 is still adding, run {{--execute --additional}} with
{{{}[1,2,4,5]{}}}.
# {{--list}} shows {{adding: 5. removing: 3.}} — 4 is gone from adding.
# Once 5 is in sync the reassignment completes: {{{}replicas=[1,2,4,5]
isr=[1,2,5]{}}}, no reassignment in progress.
Expected: step 3 shows {{adding: 4,5. removing: 3.}} and the reassignment stays
in progress until 4 is also in sync.
The override case is only correct today when the new adding set happens to be
empty (the builder leaves the old one in place) or when the RF-reduction guard
{{adding.size() < removing.size()}} blocks completion, which masks the bug for
shrinking overrides.
ZooKeeper-mode controllers computed the new adding set relative to the original
replica set, so this is a KRaft regression.
Fix: derive the new adding set as {{{}target − (currentReplicas −
currentAddingReplicas){}}}, so a replica that was still being added by the
previous reassignment stays in {{addingReplicas}} if the new target keeps it.
> KRaft controller drops still-adding replicas from addingReplicas when an
> in-progress reassignment is overridden
> ---------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-21419
> URL: https://issues.apache.org/jira/browse/KAFKA-21419
> Project: Kafka
> Issue Type: Bug
> Components: controller
> Affects Versions: 4.0.0, 4.1.0, 4.2.0, 4.3.0
> Reporter: Haifeng Chen
> Priority: Major
>
> When a partition reassignment is still in progress and a new reassignment for
> the same partition is submitted (for example with
> {{{}kafka-reassign-partitions.sh --execute --additional{}}}, which Cruise
> Control does routinely), the KRaft controller computes the new
> {{addingReplicas}} as {{{}target − currentReplicas{}}}. {{currentReplicas}}
> already contains the replicas the first reassignment is still adding, so any
> of them that are also in the new target are silently promoted to plain
> replicas even though they have not joined the ISR yet.
> The override then completes as soon as the replicas it newly adds are in
> sync, leaving the partition with a replica that is out of the ISR and that no
> reassignment accounts for: {{{}-{}}}list shows nothing in flight, {{-verify}}
> reports success, under-replicated partitions stay non-zero until the replica
> happens to catch up, and the removing replica is dropped while the partition
> has fewer in-sync replicas than it looks.
> Reproduction (any KRaft cluster):
> # Partition on {{{}[1,2,3]{}}}; reassign to {{[1,2,4]}} and hold broker 4
> out of the ISR (e.g. 1 B/s follower throttle). {{--list}} shows {{adding: 4.
> removing: 3.}}
> # While 4 is still adding, run {{--execute --additional}} with
> {{{}[1,2,4,5]{}}}.
> # {{--list}} shows {{adding: 5. removing: 3.}} — 4 is gone from adding.
> # Once 5 is in sync the reassignment completes: {{{}replicas=[1,2,4,5]
> isr=[1,2,5]{}}}, no reassignment in progress.
> Expected: step 3 shows {{adding: 4,5. removing: 3.}} and the reassignment
> stays in progress until 4 is also in sync.
> The override case is only correct today when the new adding set happens to be
> empty (the builder leaves the old one in place) or when the RF-reduction
> guard {{adding.size() < removing.size()}} blocks completion, which masks the
> bug for shrinking overrides.
> ZooKeeper-mode controllers computed the new adding set relative to the
> original replica set, so this is a KRaft regression.
> Fix: derive the new adding set as {{{}target − (currentReplicas −
> currentAddingReplicas){}}}, so a replica that was still being added by the
> previous reassignment stays in {{addingReplicas}} if the new target keeps it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
