jsancio commented on PR #12274:
URL: https://github.com/apache/kafka/pull/12274#issuecomment-1183190708
> maybe we should allow the lastAppliedOffset to be behind by A records (or
time) to unfence the broker.
@dengziming Here are my thoughts
### Improve logic for unfencing brokers
How about the controller unfence the broker when the broker's high-watermark
has reached the broker registration record for that broker? The a controller
first registers for a given incarnation, the controller write a broker
registration record. The controller can remember this offset as returned by the
raft client. The controller can unfence the broker when the broker's
high-watermark is greater than this registration offset.
### Propagate the HWM to the replicas as quickly as possible.
I think that the solution above it would allow us to de-prioritize this.
Here are my observations anyways.
Looking at the KafkaRaftClient implementation we would have to have an index
for both the fetch offset and the last sent high-watermark for that replica.
Another issue here is that we changed the KafkaRaftManager so that it
doesn't set the replica id when it is an observer/broker. Since the HWM is not
part of the Fetch request the leader would have to keep track of this in the
LeaderState
```
val nodeId = if (config.processRoles.contains(ControllerRole)) {
OptionalInt.of(config.nodeId)
} else {
OptionalInt.empty()
}
```
We would need to find a better solution for
https://issues.apache.org/jira/browse/KAFKA-13168 or improve the FETCH request
so that it includes the HWM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
