+1 (binding) Thanks Ethan, Stephen, Zita, and everyone who worked on this.
I found two issues that may need addressing, but I’m fine with filing follow-up Jiras and handling them after the merge. 1. HDDS-16025 exposed CompleteFinalizeUpgrade through client dispatch, but its handler [1] bypasses admin authorization and the normal finalization prerequisites. IMO preExecute() should be overridden to reject external requests. 2. HDDS-14671 removed HEALTHY_READONLY recovery, and HDDS-15034 added the finalization counter. A previously registered old datanode can resume heartbeats without restarting after SCM finalizes and become HEALTHY again. The version-rejection branch [2] only logs and returns, and the counter [3] treats matching datanode apparent/software versions as finalized even when both are older than SCM. Incompatible nodes should receive ReregisterCommand before refreshing their heartbeat timestamp, and finalized counts should require SCM’s software version. Thanks, Siyao [1] https://github.com/apache/ozone/blob/a9a64dbed31ba1dfee9057b7c8ddd42acb948b2f/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/upgrade/OMCompleteFinalizeUpgradeRequest.java#L68 [2] https://github.com/apache/ozone/blob/a9a64dbed31ba1dfee9057b7c8ddd42acb948b2f/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java#L806 [3] https://github.com/apache/ozone/blob/a9a64dbed31ba1dfee9057b7c8ddd42acb948b2f/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java#L185 On Tue, Oct 6, 2026 at 3:30 PM Ethan Rose <[email protected]> wrote: > Hi all, thanks for voting. All the content is ready for merge at this time: > > - master has been merged into the ZDU branch with green CI at commit > a99fabb9dea89b1d3961bd1eccbabee13f2047f7 today > - Design doc and user doc have been updated based on initial reviews. > - pull/11354 <https://github.com/apache/ozone/pull/11354> is merged > > Still outstanding: > > - +1s on the design document <https://github.com/apache/ozone/pull/9664 > >, > which should be merged with the feature branch > - Ideally at least one more binding PMC +1 from someone who did not work > on the feature directly. > - Currently we have Stephen as part of the feature, and Uma and Ayush > from outside > > We can leave the dev <https://github.com/apache/ozone-site/pull/560> and > user <https://github.com/apache/ozone-site/pull/545> docs open for reviews > for a bit after the branch merges in case people want more time to review > those. I especially want to call people's attention to the dev docs since > those will need to be understood by everyone once ZDU is merged. > > I will keep updating the feature branch from master until we are ready to > merge it in. > > Ethan > > On Mon, Oct 5, 2026 at 5:24 AM Ayush Saxena <[email protected]> wrote: > > > +1 > > > > -Ayush > > > > > On 5 Oct 2026, at 2:43 PM, Zita Dombi <[email protected]> wrote: > > > > > > +1 (non binding) to merge. Disclaimer: I worked on this feature > > > (implementation and code review) > > > > > > Zita > > > > > > Stephen O'Donnell via dev <[email protected]> ezt írta (időpont: > > 2026. > > > okt. 5., H, 11:05): > > > > > >> Summit, > > >> > > >> I think your questions are covered in the design from this part > > >> > > >> > > > https://github.com/apache/ozone/pull/9664/changes#diff-93aadee36203b3e50605117504fae31353edba5844c084e8169fde778c1c1a1cR239 > > >> > > >> Datanodes will continue to be available for writes during finalize. In > > >> short they will operate at the lowest version in the pipeline and any > > >> blocks currently being written will be written / committed at the > > earlier > > >> version. > > >> > > >> Stephen. > > >> > > >> > > >> On Mon, Oct 5, 2026 at 4:13 AM Sumit Agrawal via dev < > > [email protected] > > >>> > > >> wrote: > > >> > > >>> +1 for merge. > > >>> > > >>> Pipeline can exist in mixed version of DN (old and new). Design doc > do > > >> not > > >>> provide much clarity about the behavior: > > >>> - Weather write is supported in this case? allocating new blocks and > > >>> pipeline > > >>> - How about pipeline / block already assigned for write? Client will > > not > > >> be > > >>> aware of changes. > > >>> > > >>> Since SCM is already finalied and DN finalization is in progress > (which > > >> can > > >>> take 1-2 HB for few set of DNs where SCM will know). So there may be > a > > >>> delay and DN upgrade may also take time. > > >>> > > >>> - How about DN supporting read and write operation when finalization > is > > >> in > > >>> progress? is it blocking or request will be rejected by DNs ? > > >>> > > >>> > > >>> On Thu, Oct 1, 2026 at 7:46 PM Andrey Yarovoy via dev < > > >>> [email protected]> > > >>> wrote: > > >>> > > >>>> +1 for merge. > > >>>> > > >>>> On Thu, Oct 1, 2026 at 10:01 AM Uma Maheswara Rao Gangumalla < > > >>>> [email protected]> wrote: > > >>>> > > >>>>> Great work. Thank you Ethan and all others for working on this. > > >>>>> > > >>>>> +1 for the merge. > > >>>>> > > >>>>> Regards, > > >>>>> Uma > > >>>>> > > >>>>>> On Mon, Sep 28, 2026 at 4:02 PM Ethan Rose <[email protected]> > > wrote: > > >>>>>> > > >>>>>>> Hi Ozone devs, > > >>>>>>> > > >>>>>>> This is a vote thread to merge the zero downtime upgrade feature > > >>>>> branch ( > > >>>>>>> HDDS-14496-zdu) into master, which will enable zero downtime > > >>> upgrades > > >>>>>> from > > >>>>>>> the first release containing this feature to all future releases. > > >>>>> Please > > >>>>>>> familiarize yourself with the following documents when voting. > All > > >>> docs > > >>>>> PRs > > >>>>>> remain open so we can address questions and comments as they arise > > >>>> during > > >>>>>> the voting process. > > >>>>>> > > >>>>>> - Design document <https://github.com/apache/ozone/pull/9664> > > >>>>>> - Branch merge checklist < > > >>>>> https://github.com/apache/ozone-site/pull/562 > > >>>>>>> > > >>>>>> - User Docs <https://github.com/apache/ozone-site/pull/545> > > >>>>>> - Developer docs <https://github.com/apache/ozone-site/pull/560 > > >>> : > > >>>>> This > > >>>>>> guide covers how to handle compatibility for zero downtime > > >> upgrade > > >>>> and > > >>>>>> should be understood by all developers and reviewers. > > >>>>>> > > >>>>>> Summary of changes: New Requirements for Developers > > >>>>>> > > >>>>>> *Once this branch is merged, all further commits must be ZDU > > >>> compatible > > >>>>> as > > >>>>>> outlined in the developer guide*. > > >>>>>> > > >>>>>> The new versioning framework created on this branch provides tools > > >> to > > >>>>>> safely incorporate incompatible changes, but it does not > > >>> automatically > > >>>>>> resolve them. That remains the responsibility of developers and > > >>>>> reviewers. > > >>>>>> Improved Developer Experience > > >>>>>> > > >>>>>> Each component now uses a single ComponentVersion to track all > > >>>>> incompatible > > >>>>>> changes across disk and network as outlined in the design doc. > > >>>> Developers > > >>>>>> no longer need to reason about whether their incompatible change > > >>>>> requires a > > >>>>>> LayoutFeature, ComponentVersion, or both. As part of this change, > > >> the > > >>>>>> internal upgrade framework was rewritten and exposes a simpler API > > >> to > > >>>>>> developers. This includes strongly typed component versions (with > > >>>> integer > > >>>>>> conversion deferred until serialization), and an isSupportedBy > > >> method > > >>>> to > > >>>>>> handle all version comparisons. > > >>>>>> Improved Admin Experience > > >>>>>> > > >>>>>> To ensure finalization proceeds in the correct order as outlined > in > > >>> the > > >>>>>> design document, admins no longer have to finalize OM and SCM > > >>>>> separately. A > > >>>>>> single ozone admin upgrade finalize command triggers asynchronous > > >>>>>> finalization throughout the cluster in the defined order. A single > > >>>> status > > >>>>>> endpoint can be queried by ozone admin upgrade status, and clients > > >>> can > > >>>>>> trigger and block on finalization with one ozone admin upgrade > > >>> finalize > > >>>>>> --wait command, which handles polling of the status endpoint by > the > > >>>>> client > > >>>>>> automatically and is idempotent. > > >>>>>> > > >>>>>> Additionally, a Grafana dashboard has been added to provide a > heads > > >>> up > > >>>>> view > > >>>>>> of all components during an upgrade. It includes filtering to zoom > > >> in > > >>>> on > > >>>>> a > > >>>>>> particular area and aggregates across Datanodes to handle large > > >>>> clusters. > > >>>>>> Removed Prepare For Upgrade > > >>>>>> > > >>>>>> The "prepare for upgrade" command which put OMs in a read-only > mode > > >>>>> before > > >>>>>> an upgrade is no longer required. The CLI has been left as a no-op > > >>> for > > >>>>>> compatibility with older upgrade scripts. See the developer guide > > >>>> linked > > >>>>>> above for instructions to handle incompatible changes to OM write > > >>>>> requests. > > >>>>>> Work In Progress > > >>>>>> > > >>>>>> There is some ongoing work we will continue in parallel with the > > >>> merge > > >>>>> vote > > >>>>>> and finish before the branch is merged: > > >>>>>> > > >>>>>> - We are currently merging the latest master branch into the > > >>> feature > > >>>>>> branch, resolving conflicts, and running it through CI. The > > >> final > > >>>> hash > > >>>>>> for > > >>>>>> merge will be shared here when ready. > > >>>>>> - The ZDU design doc will be updated based on the latest copilot > > >>>>> review > > >>>>>> and other minor deviations identified from the resulting > > >>>>> implementation. > > >>>>>> - A PR to add a missed admin check on the ozone admin upgrade > > >>> status > > >>>>>> command is in flight: > > >> https://github.com/apache/ozone/pull/11354 > > >>>>>> > > >>>>>> Future Work > > >>>>>> > > >>>>>> As mentioned in the merge checklist, the current OM request > > >>> versioning > > >>>>>> framework on master was left intact on the ZDU branch. However, > the > > >>>>> opt-in > > >>>>>> annotation based approach does not suit the new ZDU requirements > > >>> where > > >>>>>> every new request needs to be versioned. Dev work has started on a > > >>> new > > >>>>>> framework, but the change is large and was deliberately saved for > > >>>> master > > >>>>>> after the branch merge so that it can be reviewed by a wider > > >>> audience. > > >>>>>> > > >>>>>> Additionally, we will be investigating static analysis and AI > > >> skills > > >>> to > > >>>>>> flag potentially incompatible changes during code reviews and the > > >>>> release > > >>>>>> process. > > >>>>>> ------------------------------ > > >>>>>> > > >>>>>> Thanks to Stephen, Zita, and Roland who also worked on the > > >>> development > > >>>> of > > >>>>>> this feature and to everyone who shared inputs on the design. > > >>>>>> > > >>>>>> We will leave the vote thread open for at least 7 days, although > > >> more > > >>>>> time > > >>>>>> may be required for developers to familiarize themselves with the > > >> new > > >>>> ZDU > > >>>>>> requirements. > > >>>>>> > > >>>>>> - Ethan > > >>>>>> > > >>>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Thanks, > > >>>> Andrey. > > >>>> > > >>> > > >>> > > >>> -- > > >>> *Sumit Agrawal* | Senior Staff Engineer > > >>> cloudera.com <https://www.cloudera.com> > > >>> [image: Cloudera] <https://www.cloudera.com/> > > >>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > >>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > >> Cloudera > > >>> on LinkedIn] <https://www.linkedin.com/company/cloudera> > > >>> ------------------------------ > > >>> > > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
