Mohammad Reduwan created KAFKA-21462:
----------------------------------------

             Summary: Adaptive partitioner sends records to empty partitions on 
a broker that partitioner.availability.timeout.ms marked unavailable
                 Key: KAFKA-21462
                 URL: https://issues.apache.org/jira/browse/KAFKA-21462
             Project: Kafka
          Issue Type: Bug
          Components: clients, producer 
    Affects Versions: 3.3.2, 3.3.1, 3.3.0
            Reporter: Mohammad Reduwan


When partitioner.availability.timeout.ms is enabled empty partitions on a slow 
or unresponsive broker incorrectly bypass the availability check. As a result, 
the producer continues routing new traffic to a degraded broker instead of 
avoiding it.

1. Problem Description
With adaptive partitioning enabled (the default) and an availability timeout 
set:
*  If a broker cannot process produce requests for longer than 
partitioner.availability.timeout.ms, its partitions should be treated as 
unavailable.
* The Bug: Only partitions that currently have a queued batch are properly 
excluded. Partitions with an empty deque on the same slow broker remain in the 
load stats.
* Because the adaptive partitioner weights partitions by the inverse of their 
queue size, an empty queue makes the partition look like one of the best 
choices. Consequently, keyless records (or records with 
partitioner.ignore.keys=true) are wrongly routed to the unavailable broker.

2. Root Cause
In RecordAccumulator#partitionReady:
* Skipped Check: A partition is added to the load stats before checking its 
deque. If the deque is empty, the code hits a continue statement, completely 
bypassing the availability timeout check that runs further down.
* Slot Reusability Bug: When an excluded partition gives up its slot 
(--queueSizesIndex), a subsequent empty partition can reuse that slot without 
resetting queueSizes. This causes the empty partition to incorrectly inherit 
the queue size of the previously excluded partition instead of registering as 0.

(This logic dates back to KAFKA-10888 / KIP-794 and was retained during the 
refactoring in KAFKA-20917).

3. Steps to Reproduce
* Set up a test scenario matching testAdaptiveBuiltInPartitioner (Node 0 leads 
Partitions 0 and 1; Node 1 leads Partition 2; availability timeout = 100ms).
* Append 1 batch to Partition 0, 1 to Partition 1, and 3 to Partition 2.
* Clear Partition 1's deque (simulating a drained in-flight request).
* Mark Node 0 as not drainable for 200ms, then call ready().
* Append 30 large records with UNKNOWN_PARTITION.
* Expected Result: No records should go to Partitions 0 or 1.
* Actual Result: Partition 0 is correctly excluded, but 24 out of 30 records 
are incorrectly sent to Partition 1 (on the slow Node 0).

4. Impact
* Failure Loop: During a broker slowdown, every empty partition led by that 
broker attracts new batches. These batches cannot be sent and eventually fail 
with a TimeoutException after delivery.timeout.ms. This empties the partitions 
again, causing the cycle to repeat continuously while the broker remains 
degraded.
* Feature Failure: The configuration specifically designed to protect producers 
from slow brokers ends up feeding them more traffic.

5. Proposed Fix
* Move the availability check to happen before the deque is inspected (right 
when the leader is looked up for load stats).
* Explicitly record a queue size of 0 for partitions with empty deques.
* Add a regression test alongside testAdaptiveBuiltInPartitioner to cover empty 
partitions on unavailable nodes.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to