Arafat Khan created HDDS-16939:
----------------------------------
Summary: Recon drops replica history for departed datanodes on OM
full reprocess
Key: HDDS-16939
URL: https://issues.apache.org/jira/browse/HDDS-16939
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Recon
Reporter: Arafat Khan
Fix For: 2.3.0
Recon's container replica history is meant to be a durable record of every
datanode that has ever held a replica of a container, including ones that are
dead or no longer hold it. It is served by *GET
/api/v1/containers/\{id}/replicaHistory*
Once a datanode leaves the replica set, its record is lost at the next OM full
reprocess.
*Repro* (verified on a 4-datanode docker-compose cluster with
dead.node.interval=45s):
# Write a key, resolve its container. Replicas on dn-1, dn-3, dn-4. GET
replicaHistory returns 3 entries.
# Stop dn-4. Once SCM declares it dead and re-replicates to dn-2, GET
replicaHistory returns 4 entries, correctly still including dn-4.
# POST /api/v1/triggerdbsync/om/reinit to force a full reprocess.
# GET replicaHistory returns 3: dn-1, dn-2, dn-3. dn-4 is gone.
SCM reported the identical live replica set before and after step 3, so the
reprocess was the only variable. Repeating the reinit when history already
equals the live set loses nothing, which confirms the loss is specifically the
departed entries.
*Root cause:*
History is held in two places and the API returns the union: the
REPLICA_HISTORY_V2 column family in the Recon RocksDB (recon-container-key.db),
and the in-memory replicaHistoryMap.
When a replica departs, ReconContainerManager.removeContainerReplica flushes
its record to the column family and then removes it from memory (lines
378-380). From that point the column family is its only copy.
An OM full reprocess builds a staging DB via
ReconDBProvider.getStagedReconDBProvider(), which creates a brand new empty
RocksDB rather than a copy. Only OM-derived tasks populate it, and none of them
write replica history, so REPLICA_HISTORY_V2 stays empty.
ReconTaskControllerImpl:307 then calls replaceStagedDb(), which renames the
live DB to .backup and moves the empty staging DB into its place.
Nothing clears the column family. There is no clear or delete on it anywhere,
only put and get. The database file it lives in is discarded and a different
file takes its name.
The in-memory map survives the swap, so live replicas are unaffected and the
loss is invisible in normal operation. What the swap destroys is exactly the
entries present in the column family but not in memory, which is the departed
datanodes. Of the eight RocksDB column families, seven are rebuilt from the OM
snapshot; REPLICA_HISTORY_V2 is the only one sourced from the SCM/datanode
side, so it is the only one with nothing to refill it.
This is a regression from HDDS-13553. Before it, the reprocess truncated three
named tables in place (containerKeyTable, keyContainerTable,
containerKeyCountTable) inside the existing DB, leaving replica history
untouched. That method still only clears those three; the damage comes from the
staging swap added alongside it.
*Impact:*
getAllContainerHistory/getLatestContainerHistory back four ContainerEndpoint
APIs: replicaHistory (:335), /containers/missing (:366), /containers/unhealthy
(:474), quasi-closed (:1024).
Missing containers is worst. Such a container has no live replicas, so memory
holds nothing for it and the column family was the only source of which
datanodes last held it. After a reprocess the report still lists the container
but with an empty datanode list, losing exactly what an operator needs during a
data-loss investigation.
*Fix:*
Carry REPLICA_HISTORY_V2 across the swap, mirroring the preservedNodes pattern
already used in ReconStorageContainerManagerFacade.initializeNewRdbStore() for
the NODES table. Copy at the top of replaceStagedDb() where both stores are
still open, streaming live to staged with a table iterator and a BatchOperation.
Related defects in the same methods, can be split out:
# batchStoreContainerReplicaHistory blind-puts the in-memory map, so the
shutdown flush clobbers departed datanodes on every graceful restart. Should
merge.
# upsertContainerHistory swallows IOException at DEBUG while
removeContainerReplica evicts from memory regardless, so a failed flush loses
the record silently.
# firstSeenTime resets on restart and then shadows the correct stored value,
because the merge at :411 is in-memory-wins. It should take the earlier of the
two.
*Test gap:*
Replica history is covered only by
TestReconContainerManager.testUpdateAndRemoveContainerReplica, which asserts
against the in-memory map and never runs a reprocess. The staging swap has
several tests, none touching replica history.
Suggested regression test, unit level, no cluster: persist history for three
datanodes, remove one, run a staging swap, assert getAllContainerHistory still
returns all four. Fails today.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
