This is an automated email from the ASF dual-hosted git repository.
errose28 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ozone-site.git
The following commit(s) were added to refs/heads/master by this push:
new 2fe40b872c HDDS-16022. Branch merge checklist for ZDU (#562)
2fe40b872c is described below
commit 2fe40b872caef3cdfad351aa0aaaa5de95ec976a
Author: Ethan Rose <[email protected]>
AuthorDate: Thu Oct 8 19:34:12 2026 -0400
HDDS-16022. Branch merge checklist for ZDU (#562)
---
.../03-merged-branches/19-hdds-14496-zdu.md | 107 +++++++++++++++++++++
1 file changed, 107 insertions(+)
diff --git
a/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md
b/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md
new file mode 100644
index 0000000000..4a2afc9438
--- /dev/null
+++
b/docs/08-developer-guide/04-project/01-git/03-feature-branches/03-merged-branches/19-hdds-14496-zdu.md
@@ -0,0 +1,107 @@
+# HDDS-14496: Zero Downtime Upgrade
+
+Epic: [HDDS-14496](https://issues.apache.org/jira/browse/HDDS-14496)
+Feature branch: https://github.com/apache/ozone/tree/HDDS-14496-zdu
+
+## 1. Builds/intermittent test failures
+
+Currently no intermittent test failures are observed on the branch. The
following integration tests were run 400 times and passed on the branch:
+
+- `TestDatanodeCurrentVersionEndToEnd`
+- `TestOMUpgradeFinalization`
+- `TestBlockDeletionService`
+- `TestUpgradeFinalizationWithOldClients`
+- `TestScmHAFinalization`
+ - Previously flaky on the branch, but fixed by
https://github.com/apache/ozone/pull/11200
+
+Additionally, `TestScmDataDistributionFinalization` was marked flaky on master
but fixed on the ZDU branch by https://github.com/apache/ozone/pull/11228
+
+AI identified test stabilization changed that were proactively applied in
https://github.com/apache/ozone/pull/11258 as well.
+
+## 2. Documentation
+
+- Documentation for admins to execute ZDU is open at
https://github.com/apache/ozone-site/pull/545
+- Documentation for developers to maintain upgrade compatibility while working
on features is open at https://github.com/apache/ozone-site/pull/560
+
+Both of these PRs are currently works in progress and will continue to be
updated during the branch review.
+
+## 3. Design, attached the docs
+
+Design document is open at https://github.com/apache/ozone/pull/9664. It has
been updated as the feature was developed, but we will do a final check for
consistency with the implementation before it is merged with the branch.
+
+## 4. S3 compatibility
+
+ZDU happens within an Ozone cluster and does not alter any user facing S3
APIs. When doing a rolling upgrade of S3 gateways, there will be a period where
the gateways are in mixed versions and may support different sets of APIs. If
this is not desirable, a load balancer and upgrade order must be configured
outside of Ozone to direct all traffic to gateways in one version while another
set is upgraded.
+
+## 5. Docker-compose / Acceptance tests
+
+A new `rolling` upgrade acceptance test execution has been added in addition
to the existing `non-rolling` flow. Due to the excessive amount of time taken
to restart containers one-by-one in the test environment, `rolling` executes as
follows:
+
+1. Rolling upgrade of all services from old to new version
+2. Non-rolling downgrade of all services from new to old version
+3. Non-rolling upgrade of all services from old to new version
+
+The initial rolling upgrade covers the same set of mixed component versions as
the downgrade, since restarts for downgrade must be done in reverse order of
the upgrade. Therefore we still get coverage of all component version
interactions through just the initial rolling upgrade.
+
+This simplification allows the execution of the rolling upgrade suite to
finish in [20
minutes](https://github.com/apache/ozone/actions/runs/33115384634/job/98781266395),
compared to doing a rolling restart for all three steps which took [1.5
hours](https://github.com/errose28/ozone/actions/runs/32282383715/job/96194185059).
Note that the original non-rolling upgrade suite finished in about [15
minutes](https://github.com/apache/ozone/actions/runs/32365620238/job/96418094876).
Since acce [...]
+
+Since rolling upgrade requires the initial version to support the feature, it
can't actually be run yet since there is currently no supported release. On the
branch we tested it by using the same version for old and new, basically doing
a rolling restart to ensure the test was runnable. When the branch is merged,
we will continue to run only the existing non-rolling upgrade tests until the
first ZDU enabled release goes out.
+
+## 6. Support of containers / Kubernetes
+
+No additions or changes in existing support.
+
+## 7. Coverage / Code quality
+
+- [Master has 79% code
coverage](https://sonarcloud.io/component_measures?metric=coverage&view=list&id=hadoop-ozone)
+- [ZDU branch has 78.7% code
coverage](https://sonarcloud.io/component_measures?metric=coverage&view=list&branch=HDDS-14496-zdu&id=hadoop-ozone)
+
+## 8. Build time
+
+- Latest successful ZDU CI runs finished in [1 hour 26
minutes](https://github.com/apache/ozone/actions/runs/34383195672) and [1 hour
30 minutes](https://github.com/apache/ozone/actions/runs/35762034464)
+- This is in line with recent master CI runs which show executions ranging
from [1 hour 26
minutes](https://github.com/apache/ozone/actions/runs/36069833598) to
+ [1 hour 31 minutes](https://github.com/apache/ozone/actions/runs/35985080804)
+
+## 9. Possible incompatible changes/used feature flag
+
+### Compatibility with existing releases
+
+This branch introduces a new unified versioning framework as outlined in the
design doc, where a single version per component is used to track all
compatibility concerns. The change from the old to new framework requires
compatibility handling from the framework itself, so it takes effect after the
new `ZDU` component version added on this branch is finalized.
+
+Additionally, new simplified CLIs to start and monitor upgrade finalization
have been added, and prepare for upgrade has been removed. The old CLIs have
been retained in a way that still allows old upgrade scripts to finalize a ZDU
enabled cluster.
+
+### Compatibility with future releases
+
+From the point this branch is merged onward (component version `ZDU`), all
commits must maintain compatibility with zero-downtime upgrade. Instructions
for developers to maintain this are added to the website in
https://github.com/apache/ozone-site/pull/560. There are multiple initiatives
that will be investigated after the branch merge to make this process easier:
+
+- Static analysis to comment on PRs and notify reviewers when potentially
incompatible parts of the code are changed
+- An AI skill that can inspect code diff between two releases and flag
potentially unaddressed incompatible changes before they are shipped
+- A new OM request versioning framework that better aligns with the more
strict requirements of ZDU on OM requests.
+
+There are no changes to our external client/server cross-compatibility
guarantees as part of this feature.
+
+## 10. Third-party dependencies/License changes
+
+There are no third party dependencies introduced by this feature.
+
+## 11. Performance
+
+With ZDU, checking versions for feature compatibility is still a constant time
operation and should not change request processing performance. Finalization
still executes as a single Ratis request and upgrade actions are still required
to be constant time, which is the same as master currently.
+
+Upgrade finalization no longer requires closing all write pipelines, which
should improve performance during finalization compared to non-rolling upgrades.
+
+## 12. Security considerations
+
+Two new user facing APIs were added which both have admin checks:
+
+- [finalize
upgrade](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/upgrade/OMFinalizeUpgradeRequestBase.java#L69)
+- [query upgrade
status](https://github.com/apache/ozone/blob/709ac02990fe3a7a6fd99a781df53e95e33d2bfd/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L3781)
+
+These commands forward to internal APIs that OM invokes on SCM through the
[`ScmBlockLocationProtocol`](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java#L45),
which is restricted to `hdds.scm.kerberos.principal` the same as existing
operations like block deletion.
+
+Internal APIs were added in OM and SCM to check that all peers have been
upgraded before allowing finalization to proceed:
+
+- `StorageContainerLocationProtocol#getPeerUpgradeStatus`
+ - Restricted to
[`hdds.scm.kerberos.principal`](https://github.com/apache/ozone/blob/81fd3c45d01b57d2c2999b7c05f6784b6b339ab1/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java#L60)
+- `OMAdminProtocol#getPeerUpgradeStatus`
+ - Restricted to
[`ozone.om.kerberos.principal`](https://github.com/apache/ozone/blob/53f472a7c04f58a021625cdc6f060bf629e9d4a7/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OMAdminProtocol.java#L31)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
