Chu Cheng Li created HDDS-17365:
-----------------------------------
Summary: SCM counts a delete transaction as done for a new copy on
a datanode that applied it to an earlier replica
Key: HDDS-17365
URL: https://issues.apache.org/jira/browse/HDDS-17365
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Chu Cheng Li
*Problem*
SCM keeps the datanodes that committed a delete transaction in
{{transactionToDNsCommitMap}}. It does not keep which replica committed it. SCM
purges the transaction when every current replica is on a datanode in this set
({{allReplicasCommitted}}).
A datanode can lose its replica after it commits the transaction. For example,
the replica becomes UNHEALTHY and SCM deletes it, or a volume fails. Later the
datanode can get a new copy of the container. If the copy comes from a replica
that did not apply the transaction yet, the new copy holds the deleted blocks.
SCM still has the datanode in the set. Thus, SCM does not send the transaction
to this datanode again. SCM purges the transaction when the other replicas
commit it.
The check for a pending replica ADD from HDDS-16144 does not prevent this. It
expects that a completed copy is in the replica set, so the purge waits for an
ACK from its datanode. Here the datanode is already in the set from its earlier
replica.
*Effect*
The new copy keeps the deleted blocks. Nothing deletes them later.
*Fix*
Options:
* When SCM adds a replica on a datanode, remove this datanode from the
committed sets of the pending transactions of the container.
* Or keep the commit per replica, not per datanode.
*How found*
The SCM simulation (HDDS-16627) checks each purge against the replicas on the
datanodes. In 5 of 100 single-SCM runs, SCM purged a transaction while a new
copy on a datanode in the committed set held the blocks (38 purges in total).
No unit test yet.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
