Lucas Brutschy created KAFKA-16198: -------------------------------------- Summary: Reconciliation may lose partitions when topic metadata is delayed Key: KAFKA-16198 URL: https://issues.apache.org/jira/browse/KAFKA-16198 Project: Kafka Issue Type: Bug Components: clients, consumer Reporter: Lucas Brutschy Assignee: Lucas Brutschy Fix For: 3.8.0
The current reconciliation code in `AsyncKafkaConsumer`s `MembershipManager` may lose part of the server-provided assignment when metadata is delayed. The reason is incorrect handling of partially resolved topic names, as in this example: * We get assigned {{T1-1}} and {{T2-1}} * We reconcile {{{}T1-1{}}}, {{T2-1}} remains in {{assignmentUnresolved}} since the topic id {{T2}} is not known yet * We get new cluster metadata, which includes {{{}T2{}}}, so {{T2-1}} is moved to {{assignmentReadyToReconcile}} * We call {{reconcile}} -- {{T2-1}} is now treated as the full assignment, so {{T1-1}} is being revoked * We end up with assignment {{T2-1, which is inconsistent with the broker-side target assignment.}} Generally, this seems to be a problem around semantics of the internal collections `assignmentUnresolved` and `assignmentReadyToReconcile`. Absence of a topic in `assignmentReadyToReconcile` may mean either revocation of the topic partition(s), or unavailability of a topic name for the topic. Internal state with simpler and correct invariant could be achieved by using a single collection `currentTargetAssignment` which is based on topic IDs and always corresponds to the latest assignment received from the broker. During every attempted reconciliation, all topic IDs will be resolved from the local cache, which should not introduce a lot of overhead. `assignmentUnresolved` and `assignmentReadyToReconcile` are removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)