Lianet Magrans created KAFKA-20332:
--------------------------------------
Summary: Ensure app thread not collecting records for partitions
being revoked
Key: KAFKA-20332
URL: https://issues.apache.org/jira/browse/KAFKA-20332
Project: Kafka
Issue Type: Bug
Components: clients, consumer
Affects Versions: 4.2.0
Reporter: Lianet Magrans
Assignee: Lianet Magrans
Fix For: 4.3.0, 4.2.1
With the changes to not fully wait on a PollEvent on the asyncConsumer app
thread (4.2), there could be a race if records are buffered for a partition and
the partition gets revoked, with the app thread potentially collecting records
from the buffer while the partition is being revoked (ending with records being
returned to the app after commit/revocation)
This sequence: hb received revoking partitions, reconciliation identified but
not triggered in background needing commit, app thread sends AsyncPoll that
triggers the reconciliation with commit, but moves on to collect records from
the buffer (fetchable if the pending reconciliation to revoke hasn't made it to
the mark to stop fetching yet). So this could lead to records loaded into
memory on the app thread, background then completing the revocation, app thread
returning records for a revoked partition.
I expect this only affects 4.2 due to the change to not fully wait on the poll
event (we used to wait before, so would never make it to collect when there was
a reconciliation attempt, ensuring we hit the mark to stop fetching before
proceeding in the app thread).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
