hachikuji opened a new pull request #9418: URL: https://github.com/apache/kafka/pull/9418
The main purpose of this patch is to add `quorum.linger.ms` behavior to the raft implementation. This gives users a powerful knob to tune the impact of fsync. When an append is accepted from the state machine, it is held in an accumulator (similar to the producer) until the configured linger time is exceeded. This allows the implementation to amortize fsync overhead at the expense of some write latency. The patch also improves our methodology for testing performance. Up to now, we have relied on the producer performance test, but it is difficult to simulate expected controller loads because producer performance is limited by other factors such as the number of producer clients and head-of-line blocking. Instead, this patch adds a workload generator which runs on the leader after election. Finally, this patch brings us nearer to the write semantics expected by the KIP-500 controller. It makes the following changes: - Introduce `RecordSerde<T>` interface which abstracts the underlying log implementation from `RaftClient`. The generic type is carried over to `RaftClient<T>` and is exposed through the read/write APIs. - `RaftClient.append` is changed to `RaftClient.scheduleAppend` and returns the last offset of the expected log append. - `RaftClient.scheduleAppend` accepts a batch of list of records and ensures that the full set are included in a single batch. - Introduce `RaftClient.Listener` with a single `handleCommit` API which will eventually replace `RaftClient.read` in order to surface committed data to the controller state machine. Currently `handleCommit` is only used for records appended by the leader. (Note that this patch piggybacks on top of #9352. I will rebase as soon as this PR has been merged.) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
