Hi Ozone devs, This is a vote thread to merge the zero downtime upgrade feature branch ( HDDS-14496-zdu) into master, which will enable zero downtime upgrades from the first release containing this feature to all future releases. Please familiarize yourself with the following documents when voting. All docs PRs remain open so we can address questions and comments as they arise during the voting process.
- Design document <https://github.com/apache/ozone/pull/9664> - Branch merge checklist <https://github.com/apache/ozone-site/pull/562> - User Docs <https://github.com/apache/ozone-site/pull/545> - Developer docs <https://github.com/apache/ozone-site/pull/560>: This guide covers how to handle compatibility for zero downtime upgrade and should be understood by all developers and reviewers. Summary of changes: New Requirements for Developers *Once this branch is merged, all further commits must be ZDU compatible as outlined in the developer guide*. The new versioning framework created on this branch provides tools to safely incorporate incompatible changes, but it does not automatically resolve them. That remains the responsibility of developers and reviewers. Improved Developer Experience Each component now uses a single ComponentVersion to track all incompatible changes across disk and network as outlined in the design doc. Developers no longer need to reason about whether their incompatible change requires a LayoutFeature, ComponentVersion, or both. As part of this change, the internal upgrade framework was rewritten and exposes a simpler API to developers. This includes strongly typed component versions (with integer conversion deferred until serialization), and an isSupportedBy method to handle all version comparisons. Improved Admin Experience To ensure finalization proceeds in the correct order as outlined in the design document, admins no longer have to finalize OM and SCM separately. A single ozone admin upgrade finalize command triggers asynchronous finalization throughout the cluster in the defined order. A single status endpoint can be queried by ozone admin upgrade status, and clients can trigger and block on finalization with one ozone admin upgrade finalize --wait command, which handles polling of the status endpoint by the client automatically and is idempotent. Additionally, a Grafana dashboard has been added to provide a heads up view of all components during an upgrade. It includes filtering to zoom in on a particular area and aggregates across Datanodes to handle large clusters. Removed Prepare For Upgrade The "prepare for upgrade" command which put OMs in a read-only mode before an upgrade is no longer required. The CLI has been left as a no-op for compatibility with older upgrade scripts. See the developer guide linked above for instructions to handle incompatible changes to OM write requests. Work In Progress There is some ongoing work we will continue in parallel with the merge vote and finish before the branch is merged: - We are currently merging the latest master branch into the feature branch, resolving conflicts, and running it through CI. The final hash for merge will be shared here when ready. - The ZDU design doc will be updated based on the latest copilot review and other minor deviations identified from the resulting implementation. - A PR to add a missed admin check on the ozone admin upgrade status command is in flight: https://github.com/apache/ozone/pull/11354 Future Work As mentioned in the merge checklist, the current OM request versioning framework on master was left intact on the ZDU branch. However, the opt-in annotation based approach does not suit the new ZDU requirements where every new request needs to be versioned. Dev work has started on a new framework, but the change is large and was deliberately saved for master after the branch merge so that it can be reviewed by a wider audience. Additionally, we will be investigating static analysis and AI skills to flag potentially incompatible changes during code reviews and the release process. ------------------------------ Thanks to Stephen, Zita, and Roland who also worked on the development of this feature and to everyone who shared inputs on the design. We will leave the vote thread open for at least 7 days, although more time may be required for developers to familiarize themselves with the new ZDU requirements. - Ethan
