Ethan Feng created RATIS-2605:
---------------------------------
Summary: Heartbeat AppendEntries success may incorrectly advance
leader-side matchIndex
Key: RATIS-2605
URL: https://issues.apache.org/jira/browse/RATIS-2605
Project: Ratis
Issue Type: Bug
Reporter: Ethan Feng
In the current AppendEntries success handling, the leader uses reply.nextIndex
to advance the follower's matchIndex and nextIndex for all successful
AppendEntries replies.
This is not always safe for heartbeat AppendEntries.
On the follower side, a successful heartbeat reply returns the follower's
local
state.getNextIndex(). This value describes the follower's local log tail, but
it does not prove that all entries up to nextIndex - 1 match the leader's log.
A heartbeat success only proves that the request's previousLog, if present, is
accepted by the follower.
This can corrupt the leader-side matchIndex when the follower has uncommitted
entries from an old leader.
One possible sequence is:
1. A server was previously leader and appended some uncommitted entries.
2. A new leader is elected in a higher term and appends different entries at
the same indexes.
3. The old leader becomes a follower, but still has the old uncommitted tail
locally.
4. The new leader sends heartbeat AppendEntries to that follower.
5. The follower replies SUCCESS and returns its local nextIndex.
6. The leader treats reply.nextIndex - 1 as a matched index and advances
matchIndex.
7. Later, when real AppendEntries requests hit the conflicting tail, the
follower correctly replies INCONSISTENCY because previousLog does not
match.
8. The leader-side inconsistency backoff uses matchIndex + 1 as a lower bound,
so the polluted matchIndex can prevent nextIndex from backing off to the
real common prefix.
9. The follower may remain stuck replying INCONSISTENCY instead of reaching
the
point where the conflicting tail can be truncated.
The root cause is that heartbeat SUCCESS and append SUCCESS have different
proof strength:
- AppendEntries with log entries proves the appended entries were accepted, so
reply.nextIndex can be used to advance matchIndex.
- Heartbeat AppendEntries proves only the request previousLog, not the
follower's whole local tail.
The leader should therefore avoid advancing matchIndex from the follower's
local nextIndex in heartbeat SUCCESS replies. For heartbeat SUCCESS, the
leader should advance at most to request.previousLog.index + 1. Normal append
SUCCESS can continue to use reply.nextIndex.
This issue can cause prolonged log replication stalls with repeated
INCONSISTENCY replies after leader changes, especially when an old leader has
a divergent uncommitted log tail.
Proposed fix:
Preserve the AppendEntries request metadata together with the reply. When
handling SUCCESS, use request.previousLog.index + 1 as the upper bound for
heartbeat requests, and keep using reply.nextIndex for requests that contain
log entries.
Test plan:
Add unit coverage for:
1. Heartbeat SUCCESS with previousLog advances only to previousLog.index + 1.
2. Heartbeat SUCCESS without previousLog does not advance matchIndex.
3. Append SUCCESS still advances using reply.nextIndex.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
