mmmmxa opened a new issue, #4707:
URL: https://github.com/apache/iggy/issues/4707
### Description
Every partition is a VSR consensus group. `Commit` signals are sent with a
certain frequency in order to communicate the latest commit id and signal
the liveness of the primary. When the group is idle, the `Commit` is send
roughly every 500-700 ms in order to communicate that the primary is still
working. It takes up a lot of network bandwidth while the system is doing
nothing. This scales with partition size and causes substantial CPU and
network bandwidth waste.
### Affected area / component
Clustering / replication, Wire protocol / API, Performance
### Proposed solution
1. **Batch the messages.** Each shard send heartbeats on a schedule to which
every group snaps to. This allows to put lots of commit messages into a
single
new `CommitMessageBatch` message, which reduces the traffic significantly.
2. **Heartbeat backoff.** If a heartbeat repeats (has the same view and
commit
point as the previous), the group waits longer before the next one,
increasing
until it hits a time limit set from a config.
**Wire change.** This adds a `#[repr(C)]` command and one field to
`CommitHeader`, and bumps `REPLICA_WIRE_REVISION` from 1 to 2. Please confirm
this is acceptable.
#### Results so far
### Alternatives considered
- Introduce a separate shard-level liveness signal and a separate message to
inform
backups of a `Commit` information. This requires a protocol change that is
harder
to reason about and is likely a future improvement.
- Only batch, with no back-off. Current results show that the batch is
responsible for
~83% of traffic reduction and the heartbeat backoff is ~10%. This may seem
insignificant,
but at scale this reduction is substantial.
### Contribution
- [x] I'm willing to submit a pull request to implement this feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
