mmmmxa commented on issue #4707:
URL: https://github.com/apache/iggy/issues/4707#issuecomment-6665830084
Hi @hubcio, I am close to finish with the PR so I'd like to share the spec
in order to get your feedback on the design.
1. The implementation is mostly fitting this spec, except the `CommitReply`
part, which turned out to need batching at scale too.
2. "OK" next to requirement means I have finalized it and I am 90% confident
that this should stay as is. Everything that is not labeled this way means it
is subject to change and discussion.
3. Implementation-wise everything that is OK is mostly working.
`CommitReply` is not implemented yet, so the spec may change in that regard.
# Grouping Commit messages: requirements
## Purpose
The primary of a consensus group sends Commit messages to its backups. A
Commit shows that the primary is alive and tells the backups its commit point.
A group is the metadata group or one partition group.
This work also batches the `CommitReply` frames that the backups send back
(R26 to R32).
Today every group sends its own 256-byte Commit frame to each backup. Even
with no client connected, a 3-node cluster with 500 partitions sends about 830
Commit frames every 500 ms. This work cuts that traffic in two ways:
- A shard sends all Commits that it owes one peer node on one tick in one
batch frame. Each group then costs a 40-byte entry, not a 256-byte frame.
- A group with no writes sends Commits less often, and its backups wait
longer (R20).
Terms used below:
- P is `commit_broadcast_interval` in ticks.
- A heartbeat is a Commit that a primary sends on a schedule.
- An eager Commit is a Commit that a primary sends right after its commit
point moves, to each backup that holds the newly committed op.
- A send tick is a tick on which a shard sends heartbeats.
- A member index is the position of a node inside one group. Since #4692 the
same node has a different member index in each group.
## Requirements
### Schedule - OK
- OK R1: Each shard has one send tick every P ticks. Heartbeats leave only
on send ticks.
- OK R2: When a group comes online or becomes primary, its first heartbeat
goes on the next send tick of the shard.
- OK R3: After that, a busy group sends a heartbeat on every send tick (R20).
- OK R4: The metadata group follows the same schedule as the partition
groups.
- OK R5: The schedule counts ticks, not wall-clock time, so the simulator
runs it without changes.
- OK R6: Eager Commits do not wait for a send tick. They leave in the next
batch for their peer.
### Liveness - OK
- OK R7: Prepares never replace a heartbeat. Prepares travel down a chain of
backups and can miss one.
- OK R8: Only a Commit from the primary of the backup's current view
restarts the heartbeat timeout of the backup.
- OK R9: The boot checks for `commit_heartbeat_timeout` stay: at least 2 s,
and at least 4 × `commit_broadcast_interval`.
### Busy and idle groups - OK
A group is busy while its commit point moves. After one
`commit_heartbeat_timeout` with no new commit point, it turns idle. The next
new commit point makes it busy again.
| State | Primary sends a heartbeat | Backups wait before a view change |
| ----- | ------------------------- | --------------------------------- |
| Busy | every `commit_broadcast_interval` (500 ms) |
`commit_heartbeat_timeout` (5 s) |
| Idle | every `commit_idle_heartbeat_interval` (2 s) |
`commit_idle_heartbeat_timeout` (12 s) |
The two states trade failover time against traffic. Failover time is the
time from the death of a primary to the start of a view change. In a busy
group, every second of failover stops writes, so the group keeps the short
timeout. An idle group has no writes in flight, so it sends a quarter of the
heartbeats and accepts a longer failover.
The states do not change how a commit point reaches a backup. The eager
Commit delivers it at once, and the `CommitReply` of the backup shows that it
arrived. The group therefore jumps between the two states and does not step
through intermediate intervals.
The busy state has a cost. A stall of 5 s to 12 s on the shard of the
primary starts a view change in a busy group. An idle group survives it. Reads
do not count as activity, so a group with only reads turns idle.
- OK R20: On each send tick, the primary compares the `view` and
`commit_min` of the group with its last heartbeat. When they did not change for
`commit_heartbeat_timeout`, the group turns idle. Its first idle heartbeat goes
on that same send tick, so the backups learn that the group is idle before the
longer gap starts. After that, the group sends only on the send ticks that line
up with `commit_idle_heartbeat_interval`. These ticks are the same for all
groups on the shard, so idle groups share frames.
- OK R21: Each Commit entry carries `heartbeat_state`, a one-byte enum with
three values: `Busy` (a heartbeat of a busy group), `Idle` (a heartbeat of an
idle group), and `Eager` (an eager Commit). The wire holds no interval. A
backup rejects an entry with any other value, as a bad entry (R17).
- OK R22: A Commit is newer if it has a higher `view`, or the same `view`
and a higher `timestamp_monotonic`. When a backup receives a Commit newer than
any it saw, it restarts its heartbeat timeout. If `heartbeat_state` is `Idle`,
it waits `commit_idle_heartbeat_timeout`. If the value is `Busy` or `Eager`, it
waits `commit_heartbeat_timeout`. Each node reads both timeouts from its own
configuration, so all nodes must use the same configuration.
- OK R23: Four `[cluster]` options control the two states. Their defaults
are in `core/server/config.toml`, so a configuration file does not need to
declare them:
- `commit_broadcast_interval`: 500 ms.
- `commit_heartbeat_timeout`: 5 s. This is also the quiet time before a
group turns idle. This is changed from `heartbeat_timeout` name without
backward compatibility.
- `commit_idle_heartbeat_interval`: 2 s. It must be a whole multiple of
`commit_broadcast_interval`. If the two are equal, the idle state has no effect.
- `commit_idle_heartbeat_timeout`: 12 s. A backup never waits less than
`commit_heartbeat_timeout`, so a larger `commit_heartbeat_timeout` raises this
value. The longer of the two must be at least 4 ×
`commit_idle_heartbeat_interval`.
- OK R24: The quorum window of the primary stays at
`commit_heartbeat_timeout` in both states. The quorum window is the time in
which the primary must hear `CommitReply` messages from a quorum, or step down.
It never uses the idle timeout. With the idle timeout, a primary cut off from
its backups can serve reads for up to 12 s.
### Sending - OK
- OK R10: A Commit frame (command 9) is a batch. The header holds `cluster`
and the id of the sending node once. The body holds one 40-byte entry per
group: `group`, `partition_incarnation`, `view`, `commit`,
`timestamp_monotonic`, `heartbeat_state`, and `replica`. In an entry, `replica`
is the sender's member index in that group.
- OK R11: On each tick, a shard builds one batch for each peer node, with
every Commit it owes that peer. A shard adds only the groups that it leads. If
only the metadata plane ticked, the batch still goes out.
- OK R12: A group saves its superblock before its Commit joins a batch. If
the save fails, the shard sends the other groups, and the Commits of that group
go into a later batch.
- OK R13: A batch is never split across frames. One frame holds about 1.7
million entries, and a shard runs out of memory long before it leads that many
groups. A debug assertion in the encoder guards the limit.
- OK R14: `timestamp_monotonic` grows for each group, as before.
- OK R25: After a batch send, each group records for its own member whether
the send reached a connected link.
### Receiving - all OK but may be altered by Reply part
- OK R15: If the checksum of a frame is wrong, the receiver drops the whole
frame.
- OK R16: The receiving shard passes each entry to the shard that owns its
group. No entry is lost or handled twice.
- OK R17: The receiver handles each entry like a single Commit with the same
values. A bad entry is rejected alone, and the other entries are still handled.
### Replying
Today a backup answers every accepted Commit entry with its own 256-byte
`CommitReply` frame. On an idle cluster, these replies are most of the
remaining traffic: about 2000 reply frames per 5 s in the 1000-partition
traffic test, against about 24 Commit batches. A reply carries no more than
about 32 bytes of facts.
- R26: A `CommitReply` frame (command 10) is a batch, like a Commit frame.
The header holds `cluster` and the id of the sending node once. The body holds
one fixed-size entry per reply: `group`, `partition_incarnation`, `view`,
`timestamp_monotonic`, and `replica`. The entry holds the same values as a
single `CommitReply` does today.
- R27: Each shard has a reply outbox, one queue per peer node. When the
shard accepts a Commit entry, it queues the reply and does not send it. The
shard sends the queue as one batch per peer at the end of its next tick. This
holds for the partition tick and for the metadata tick. A metadata-only tick
sends its replies too.
- R28: A reply waits at most one shard tick in the outbox. The quorum window
of the primary (R24) does not change.
- R29: The receiver handles each reply entry like a single `CommitReply`
with the same values. A wrong frame checksum drops the whole frame. A bad entry
is rejected alone, and the other entries are still handled. The receiving shard
passes each entry to the shard that owns its group, and no entry is lost or
handled twice (R16).
- R30: A shard that fails to send a reply batch drops it. The next Commit
from the primary brings a new reply.
### Rollout and metrics
- OK R18: All nodes in a cluster are upgraded together. The replica wire
revision is 2, and the handshake rejects a node on revision 1.
- OK R19: Shard metrics count the batches and entries that a shard sends and
receives.
- R32: Shard metrics count the reply batches and reply entries that a shard
sends and receives. TBD: more metrics written, e.g. the heartbeat state metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
