lucasbru opened a new pull request, #23761: URL: https://github.com/apache/kafka/pull/23761
This is the first change of a series that makes the warm-up task refiner (KAFKA-20665) compatible with clients that speak version 0 of the streams group heartbeat (4.2 and 4.3), and with a coordinator failover during which the coordinator does not know the version of any member. A warm-up task is only useful if the coordinator can observe how far its holder has restored it, which a member reports through the task offsets of a version 1 heartbeat. A version 0 member never does, so it could never be promoted from warm-up to active. This change only provides the information. The coordinator records the version of every member's last heartbeat as transient state of the streams group, with the same lifecycle as the reported task offsets: written in the heartbeat path, dropped when the member is removed, and not known again after a coordinator failover until the member heartbeats. It is deliberately not persisted, so that it can never outlive the client that sent it. From it, the refiner is handed one fact per member, whether warm-up tasks are supported, not supported, or unknown. A member that the coordinator has not heard from since it loaded the group is unknown, rather than assumed to be a new client. The refiner ignores the new input for now; the following changes use it. The coordinator additionally guards the invariant that a warm-up task never reaches a member that is not known to support them. If the intermediate assignment places a warm-up task on a member that is not supported, the coordinator logs an error and reconciles towards the target assignment instead, which revokes warm-up tasks, like it already does for an intermediate assignment that loses an active task. A member of unknown version may keep a warm-up task that its process already holds, since right after a failover every member is unknown and revoking every warm-up task would throw away restore progress. Design: https://confluentinc.atlassian.net/wiki/spaces/KSTREAMS/pages/6201639231 (KAFKA-21111). The test coverage here is unit tests for the group state, for the version reaching the refiner through the heartbeat path, and for the fallback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
