[
https://issues.apache.org/jira/browse/HDDS-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devesh Kumar Singh resolved HDDS-16939.
---------------------------------------
Resolution: Fixed
> Recon drops replica history for departed datanodes on OM full reprocess
> -----------------------------------------------------------------------
>
> Key: HDDS-16939
> URL: https://issues.apache.org/jira/browse/HDDS-16939
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Recon
> Reporter: Arafat Khan
> Assignee: Devesh Kumar Singh
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> Recon's container replica history is meant to be a durable record of every
> datanode that has ever held a replica of a container, including ones that are
> dead or no longer hold it. It is served by *GET
> /api/v1/containers/\{id}/replicaHistory*
> Once a datanode leaves the replica set, its record is lost at the next OM
> full reprocess.
> *Repro* (verified on a 4-datanode docker-compose cluster with
> dead.node.interval=45s):
> # Write a key, resolve its container. Replicas on dn-1, dn-3, dn-4. GET
> replicaHistory returns 3 entries.
> # Stop dn-4. Once SCM declares it dead and re-replicates to dn-2, GET
> replicaHistory returns 4 entries, correctly still including dn-4.
> # POST /api/v1/triggerdbsync/om/reinit to force a full reprocess.
> # GET replicaHistory returns 3: dn-1, dn-2, dn-3. dn-4 is gone.
> SCM reported the identical live replica set before and after step 3, so the
> reprocess was the only variable. Repeating the reinit when history already
> equals the live set loses nothing, which confirms the loss is specifically
> the departed entries.
> *Root cause:*
> History is held in two places and the API returns the union: the
> REPLICA_HISTORY_V2 column family in the Recon RocksDB
> (recon-container-key.db), and the in-memory replicaHistoryMap.
> When a replica departs, ReconContainerManager.removeContainerReplica flushes
> its record to the column family and then removes it from memory (lines
> 378-380). From that point the column family is its only copy.
> An OM full reprocess builds a staging DB via
> ReconDBProvider.getStagedReconDBProvider(), which creates a brand new empty
> RocksDB rather than a copy. Only OM-derived tasks populate it, and none of
> them write replica history, so REPLICA_HISTORY_V2 stays empty.
> ReconTaskControllerImpl:307 then calls replaceStagedDb(), which renames the
> live DB to .backup and moves the empty staging DB into its place.
> Nothing clears the column family. There is no clear or delete on it anywhere,
> only put and get. The database file it lives in is discarded and a different
> file takes its name.
> The in-memory map survives the swap, so live replicas are unaffected and the
> loss is invisible in normal operation. What the swap destroys is exactly the
> entries present in the column family but not in memory, which is the departed
> datanodes. Of the eight RocksDB column families, seven are rebuilt from the
> OM snapshot; REPLICA_HISTORY_V2 is the only one sourced from the SCM/datanode
> side, so it is the only one with nothing to refill it.
> This is a regression from HDDS-13553. Before it, the reprocess truncated
> three named tables in place (containerKeyTable, keyContainerTable,
> containerKeyCountTable) inside the existing DB, leaving replica history
> untouched. That method still only clears those three; the damage comes from
> the staging swap added alongside it.
> *Impact:*
> getAllContainerHistory/getLatestContainerHistory back four ContainerEndpoint
> APIs: replicaHistory (:335), /containers/missing (:366),
> /containers/unhealthy (:474), quasi-closed (:1024).
> Missing containers is worst. Such a container has no live replicas, so memory
> holds nothing for it and the column family was the only source of which
> datanodes last held it. After a reprocess the report still lists the
> container but with an empty datanode list, losing exactly what an operator
> needs during a data-loss investigation.
> *Fix:*
> Carry REPLICA_HISTORY_V2 across the swap, mirroring the preservedNodes
> pattern already used in
> ReconStorageContainerManagerFacade.initializeNewRdbStore() for the NODES
> table. Copy at the top of replaceStagedDb() where both stores are still open,
> streaming live to staged with a table iterator and a BatchOperation.
> Related defects in the same methods, can be split out:
> # batchStoreContainerReplicaHistory blind-puts the in-memory map, so the
> shutdown flush clobbers departed datanodes on every graceful restart. Should
> merge.
> # upsertContainerHistory swallows IOException at DEBUG while
> removeContainerReplica evicts from memory regardless, so a failed flush loses
> the record silently.
> # firstSeenTime resets on restart and then shadows the correct stored value,
> because the merge at :411 is in-memory-wins. It should take the earlier of
> the two.
> *Test gap:*
> Replica history is covered only by
> TestReconContainerManager.testUpdateAndRemoveContainerReplica, which asserts
> against the in-memory map and never runs a reprocess. The staging swap has
> several tests, none touching replica history.
> Suggested regression test, unit level, no cluster: persist history for three
> datanodes, remove one, run a staging swap, assert getAllContainerHistory
> still returns all four. Fails today.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
