ArafatKhan2198 opened a new pull request, #12096:
URL: https://github.com/apache/ozone/pull/12096

   ## What changes were proposed in this pull request?
   
   When Recon brings a deleted container back to match SCM, it deletes its own 
record and adds a new one. Deleting the record also deletes the list of 
datanodes that hold copies. Recon then treats the container as missing even 
though the data is still there. SCM makes the same move in place and keeps that 
list. This change makes Recon do the same.
   
   **What goes wrong**
   
   1. Recon reads a page of containers that SCM has marked deleted.
   2. Before Recon finishes that page, a datanode reports a copy that still has 
data. SCM brings the container back to QUASI_CLOSED or CLOSED.
   3. Recon finishes the old page and marks the container deleted anyway. It 
still knows which datanodes hold copies.
   4. On the next sync, SCM lists the container as alive. 
`rebuildContainerFromScm` deletes Recon's record and adds a new one from SCM's 
container info.
   5. The new record has no replicas. The next health check, every 5 minutes by 
default, marks the container MISSING. It stays MISSING until each datanode's 
next full report, up to 60 minutes by default, adds the replicas back. No data 
is lost.
   6. If adding the new record fails, the failure is only logged and the 
container is gone from Recon. An old MISSING row can still be in the 
unhealthy-containers table when no health check ran between the two syncs, for 
example after a restart. `/api/v1/containers/unhealthy` then returns HTTP 500 
for every page that includes that row, until a later sync adds the container 
back. At the default interval that is 6 hours.
   
   **Change**
   
   `rebuildContainerFromScm` still checks that SCM lists the container in that 
state. It then calls `transitionDeletingOrDeletedToTargetState`, the same 
in-place move SCM's report handler uses, instead of deleting and re-adding the 
record. The replicas stay. If the update fails, the container stays in Recon as 
DELETED with its replicas, the unhealthy-containers API keeps working, and the 
next sync tries again. Recon keeps its own container details (sequence id, key 
count, used bytes). Replica reports already keep those up to date.
   
   Recon can still mark the container deleted from the stale page. It shows 
DELETED until the next sync moves it back. Skipping a missing container in the 
unhealthy-containers API, instead of failing the page, is left as separate 
hardening.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-17061
   
   ## How was this patch tested?
   
   Ran the `TestReconSCMContainerSyncIntegration` in `:ozone-recon` and had a 
green run - 
   
   ```bash
   mvn -pl :ozone-recon test -Dtest=TestReconSCMContainerSyncIntegration 
-DskipShade -DskipRecon
   ```
   
   - `containerDeletedInReconButClosedInScmKeepsReplicas`: Recon has the 
container DELETED with one replica, and SCM lists it CLOSED. After sync the 
container is CLOSED and the replica is still there.
   - `failedCorrectionFromDeletedKeepsContainerInRecon`: the in-place update 
throws. The container stays in Recon as DELETED, with its replica.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to