Ashish Kumar created HDDS-13846:
-----------------------------------
Summary: Container CLOSED with sequence ID lower than a replica
Key: HDDS-13846
URL: https://issues.apache.org/jira/browse/HDDS-13846
Project: Apache Ozone
Issue Type: Bug
Reporter: Ashish Kumar
Assignee: Ashish Kumar
In below case SCM closes container when sequence id of container is lesser than
replica sequence id.
{{}}
{code:java}
ERROR
[node1-FixedThreadPoolWithAffinityExecutor-9-0]-org.apache.hadoop.hdds.scm.container.ContainerReportHandler:
Container CLOSED with sequence ID lower than a replica: Container #2006
(CLOSED, sid=5408) r0 (CLOSED, bcsid=6556,
origin=858ea3bc-f045-442f-a34e-6a3e3c6e38f4, non-empty) from dn
858ea3bc-f045-442f-a34e-6a3e3c6e38f4(xxxxxx), proto=containerID: 2006 state:
CLOSED used: 4364173816 keyCount: 26 readCount: 411 writeCount: 1215 readBytes:
1723858944 writeBytes: 4988468861 deleteTransactionId: 7006
blockCommitSequenceId: 6304 originNodeId:
"858ea3bc-f045-442f-a34e-6a3e3c6e38f4" replicaIndex: 0 isEmpty: false
dataChecksum: 121871427{code}
{{}}
h4. Assume below scenario:
h4. Initial State:
* Leader SCM: {{containerInfo(state=CLOSING, sequenceId=6556)}} (in memory)
* Follower SCM: {{containerInfo(state=CLOSING, sequenceId=5408)}} (in memory)
h4. When CLOSE Event Triggers on Leader:
*Step 1 - Leader Processing:*
{{}}
{code:java}
// Leader SCM:
final ContainerInfo oldInfo = containers.getContainerInfo(id);
// sequenceId=6556
containers.updateState(id, CLOSING, CLOSED); // State: CLOSING → CLOSED
transactionBuffer.addToBuffer(containerStore, id,
containers.getContainerInfo(id));
// Persists: state=CLOSED, sequenceId=6556{code}
{{}}
*Step 2 - Raft Replication:*
{{}}
{code:java}
// Raft log entry contains: CLOSE event for containerID
// NO sequenceId information in the Raft log!{code}
{{}}
*Step 3 - Follower Processing:*
{{}}
{code:java}
// Follower SCM applies the same transaction:
final ContainerInfo oldInfo = containers.getContainerInfo(id);
// sequenceId=5408
containers.updateState(id, CLOSING, CLOSED); // State: CLOSING → CLOSED
transactionBuffer.addToBuffer(containerStore, id,
containers.getContainerInfo(id));
// Persists: state=CLOSED, sequenceId=5408 (STALE){code}
{{}}
After the CLOSE operation:
* Leader DB: {{state=CLOSED, sequenceId=6556}}
* Follower DB: {{state=CLOSED, sequenceId=5408}}
Since recently log is added in HDDS-12409 it is more visible.
But eventually in follower as well sequence id is getting [updated in
memory|https://github.infra.cloudera.com/CDH/ozone/blob/020aa2248702aa7402becc7b5229dce096a153e0/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/AbstractContainerReportHandler.java#L142](In
DB it remains old).
This is wrt current code logic, as in follower sequence id update is eventual
but state is consistent across all 3 SCMs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
