Meng-Shuan Tsai created HDDS-16378:
--------------------------------------
Summary: ozone repair datanode upgrade-container-schema --dry-run
writes upgrade.complete, so the real run skips the volume
Key: HDDS-16378
URL: https://issues.apache.org/jira/browse/HDDS-16378
Project: Apache Ozone
Issue Type: Bug
Reporter: Meng-Shuan Tsai
Assignee: Meng-Shuan Tsai
This is a bug in the offline schema-upgrade repair tool.
{{ozone repair datanode upgrade-container-schema --dry-run}} does not migrate
container metadata (correct), but it still writes the permanent
{{upgrade.complete}} marker under each successful volume's {{{}hddsRootDir{}}}.
The next real run treats every marked volume as already upgraded, prints {{{}is
already upgraded, skip it.{}}}, and exits 0. Containers stay on schema V2.
h2. Root cause
The migration body has three {{isDryRun()}} guards (read-only DB, no V3
{{{}put{}}}, no schema-file rewrite):
[UpgradeContainerSchema.java
L279|https://github.com/apache/ozone/blob/341b76d655aa01978a33d34fd8dce3d8d9003394/hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/datanode/schemaupgrade/UpgradeContainerSchema.java#L279],
[L433|https://github.com/apache/ozone/blob/341b76d655aa01978a33d34fd8dce3d8d9003394/hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/datanode/schemaupgrade/UpgradeContainerSchema.java#L433],
[L463|https://github.com/apache/ozone/blob/341b76d655aa01978a33d34fd8dce3d8d9003394/hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/datanode/schemaupgrade/UpgradeContainerSchema.java#L463]
The completion marker does not. Dry-run still calls {{{}result.success(){}}},
then {{whenComplete}} creates the flag whenever {{{}e == null &&
r.isSuccess(){}}}:
[UpgradeContainerSchema.java
L307–L327|https://github.com/apache/ozone/blob/341b76d655aa01978a33d34fd8dce3d8d9003394/hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/datanode/schemaupgrade/UpgradeContainerSchema.java#L307-L327]
{code:java}
result.setResultList(resultList);
result.success();
return result;
}).whenComplete((r, e) -> {
final File file =
UpgradeUtils.getVolumeUpgradeCompleteFile(r.getHddsVolume());
if (e == null && r.isSuccess()) { // missing && !isDryRun()
try {
UpgradeUtils.createFile(file);
} catch (IOException ioe) {
error(ioe, "Failed to create upgrade complete file %s.", file);
}
}
...
});
{code}
{{isAlreadyUpgraded}} only checks that the file exists:
[UpgradeUtils.java
L125–L129|https://github.com/apache/ozone/blob/341b76d655aa01978a33d34fd8dce3d8d9003394/hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/datanode/schemaupgrade/UpgradeUtils.java#L125-L129]
That check removes the volume in {{{}execute(){}}}:
[UpgradeContainerSchema.java
L174–L188|https://github.com/apache/ozone/blob/341b76d655aa01978a33d34fd8dce3d8d9003394/hadoop-ozone/cli-repair/src/main/java/org/apache/hadoop/ozone/repair/datanode/schemaupgrade/UpgradeContainerSchema.java#L174-L188]
h2. Impact
* The one-time V2→V3 migration never happens on volumes that dry-run marked
complete. Later real runs keep skipping them (exit 0, {{info}} only).
* Scope is the datanode that ran {{{}--dry-run{}}}, for volumes that
{{{}success(){}}}-ed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
