apoorvmittal10 opened a new pull request, #23951:
URL: https://github.com/apache/kafka/pull/23951
A share fetch hands its record budget to partitions in the order they are
read, so the partition read first
takes the largest share. Two things decided that order earlier and together
they could starve a partition, hence changed the behaviour.
Earlier:
- `SharePartitionManager` rotated the fetch list by `sessionEpoch % size`.
The client bumps the epoch on both
ShareFetch and ShareAcknowledge, so in a poll-and-commit loop fetch epochs
step by two and the rotation is
constant. One unpaired request flips the parity, and with two partitions
the rotation then undoes the
session reorder on every fetch, so the same partition is read first until
the other starves.
- `ShareSessionContext` moved served partitions to the end of the session
from inside the response iterator,
in the iteration order of the response map. That map had passed through
`HashMap` copies, so when every
partition returned records the session ended up in hash order and the same
partition led every fetch.
This change removes the epoch rotation from the fetch path and makes the
session reorder deterministic from
the fetch order. The fetch order is preserved through `ShareFetchUtils`,
`ShareFetch` and `KafkaApis` with
`LinkedHashMap`s. `ShareSessionContext` reorders the session in one pass
before pruning the response:
partitions without records keep their position at the front, and partitions
with records are appended in
fetch order rotated by one, so the partition read first is read last next
time. `PartitionRotateStrategy`
is reduced to that rotation and the unused `sessionEpoch` parameter of
`fetchMessages` is dropped.
`ShareConsumerFairnessTest` adds cases for the epoch parity change, for the
first position rotating when all partitions return records with explicit and
implicit acknowledgement, and for an idle partition being read first once it
has records.
`SharePartitionManagerTest` verifies the session order across subsequent
fetches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
