Mohammad Reduwan created KAFKA-21462:
----------------------------------------
Summary: Adaptive partitioner sends records to empty partitions on
a broker that partitioner.availability.timeout.ms marked unavailable
Key: KAFKA-21462
URL: https://issues.apache.org/jira/browse/KAFKA-21462
Project: Kafka
Issue Type: Bug
Components: clients, producer
Affects Versions: 3.3.2, 3.3.1, 3.3.0
Reporter: Mohammad Reduwan
When partitioner.availability.timeout.ms is enabled empty partitions on a slow
or unresponsive broker incorrectly bypass the availability check. As a result,
the producer continues routing new traffic to a degraded broker instead of
avoiding it.
1. Problem Description
With adaptive partitioning enabled (the default) and an availability timeout
set:
* If a broker cannot process produce requests for longer than
partitioner.availability.timeout.ms, its partitions should be treated as
unavailable.
* The Bug: Only partitions that currently have a queued batch are properly
excluded. Partitions with an empty deque on the same slow broker remain in the
load stats.
* Because the adaptive partitioner weights partitions by the inverse of their
queue size, an empty queue makes the partition look like one of the best
choices. Consequently, keyless records (or records with
partitioner.ignore.keys=true) are wrongly routed to the unavailable broker.
2. Root Cause
In RecordAccumulator#partitionReady:
* Skipped Check: A partition is added to the load stats before checking its
deque. If the deque is empty, the code hits a continue statement, completely
bypassing the availability timeout check that runs further down.
* Slot Reusability Bug: When an excluded partition gives up its slot
(--queueSizesIndex), a subsequent empty partition can reuse that slot without
resetting queueSizes. This causes the empty partition to incorrectly inherit
the queue size of the previously excluded partition instead of registering as 0.
(This logic dates back to KAFKA-10888 / KIP-794 and was retained during the
refactoring in KAFKA-20917).
3. Steps to Reproduce
* Set up a test scenario matching testAdaptiveBuiltInPartitioner (Node 0 leads
Partitions 0 and 1; Node 1 leads Partition 2; availability timeout = 100ms).
* Append 1 batch to Partition 0, 1 to Partition 1, and 3 to Partition 2.
* Clear Partition 1's deque (simulating a drained in-flight request).
* Mark Node 0 as not drainable for 200ms, then call ready().
* Append 30 large records with UNKNOWN_PARTITION.
* Expected Result: No records should go to Partitions 0 or 1.
* Actual Result: Partition 0 is correctly excluded, but 24 out of 30 records
are incorrectly sent to Partition 1 (on the slow Node 0).
4. Impact
* Failure Loop: During a broker slowdown, every empty partition led by that
broker attracts new batches. These batches cannot be sent and eventually fail
with a TimeoutException after delivery.timeout.ms. This empties the partitions
again, causing the cycle to repeat continuously while the broker remains
degraded.
* Feature Failure: The configuration specifically designed to protect producers
from slow brokers ends up feeding them more traffic.
5. Proposed Fix
* Move the availability check to happen before the deque is inspected (right
when the leader is looked up for load stats).
* Explicitly record a queue size of 0 for partitions with empty deques.
* Add a regression test alongside testAdaptiveBuiltInPartitioner to cover empty
partitions on unavailable nodes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)