+1. Thanks to everyone who worked on this feature. -Sadanand
On Thu, Oct 8, 2026 at 9:36 AM Siyao Meng <[email protected]> wrote: > +1 (binding) > > Thanks Ethan, Stephen, Zita, and everyone who worked on this. > > I found two issues that may need addressing, but I’m fine with filing > follow-up Jiras and handling them after the merge. > > 1. HDDS-16025 exposed CompleteFinalizeUpgrade through client dispatch, but > its handler [1] bypasses admin authorization and the normal finalization > prerequisites. IMO preExecute() should be overridden to reject external > requests. > > 2. HDDS-14671 removed HEALTHY_READONLY recovery, and HDDS-15034 added the > finalization counter. A previously registered old datanode can resume > heartbeats without restarting after SCM finalizes and become HEALTHY again. > The version-rejection branch [2] only logs and returns, and the counter [3] > treats matching datanode apparent/software versions as finalized even when > both are older than SCM. Incompatible nodes should receive > ReregisterCommand before refreshing their heartbeat timestamp, and > finalized counts should require SCM’s software version. > > Thanks, > Siyao > > [1] > > https://github.com/apache/ozone/blob/a9a64dbed31ba1dfee9057b7c8ddd42acb948b2f/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/upgrade/OMCompleteFinalizeUpgradeRequest.java#L68 > > [2] > > https://github.com/apache/ozone/blob/a9a64dbed31ba1dfee9057b7c8ddd42acb948b2f/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java#L806 > > [3] > > https://github.com/apache/ozone/blob/a9a64dbed31ba1dfee9057b7c8ddd42acb948b2f/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java#L185 > > On Tue, Oct 6, 2026 at 3:30 PM Ethan Rose <[email protected]> wrote: > > > Hi all, thanks for voting. All the content is ready for merge at this > time: > > > > - master has been merged into the ZDU branch with green CI at commit > > a99fabb9dea89b1d3961bd1eccbabee13f2047f7 today > > - Design doc and user doc have been updated based on initial reviews. > > - pull/11354 <https://github.com/apache/ozone/pull/11354> is merged > > > > Still outstanding: > > > > - +1s on the design document < > https://github.com/apache/ozone/pull/9664 > > >, > > which should be merged with the feature branch > > - Ideally at least one more binding PMC +1 from someone who did not > work > > on the feature directly. > > - Currently we have Stephen as part of the feature, and Uma and > Ayush > > from outside > > > > We can leave the dev <https://github.com/apache/ozone-site/pull/560> and > > user <https://github.com/apache/ozone-site/pull/545> docs open for > reviews > > for a bit after the branch merges in case people want more time to review > > those. I especially want to call people's attention to the dev docs since > > those will need to be understood by everyone once ZDU is merged. > > > > I will keep updating the feature branch from master until we are ready to > > merge it in. > > > > Ethan > > > > On Mon, Oct 5, 2026 at 5:24 AM Ayush Saxena <[email protected]> wrote: > > > > > +1 > > > > > > -Ayush > > > > > > > On 5 Oct 2026, at 2:43 PM, Zita Dombi <[email protected]> wrote: > > > > > > > > +1 (non binding) to merge. Disclaimer: I worked on this feature > > > > (implementation and code review) > > > > > > > > Zita > > > > > > > > Stephen O'Donnell via dev <[email protected]> ezt írta (időpont: > > > 2026. > > > > okt. 5., H, 11:05): > > > > > > > >> Summit, > > > >> > > > >> I think your questions are covered in the design from this part > > > >> > > > >> > > > > > > https://github.com/apache/ozone/pull/9664/changes#diff-93aadee36203b3e50605117504fae31353edba5844c084e8169fde778c1c1a1cR239 > > > >> > > > >> Datanodes will continue to be available for writes during finalize. > In > > > >> short they will operate at the lowest version in the pipeline and > any > > > >> blocks currently being written will be written / committed at the > > > earlier > > > >> version. > > > >> > > > >> Stephen. > > > >> > > > >> > > > >> On Mon, Oct 5, 2026 at 4:13 AM Sumit Agrawal via dev < > > > [email protected] > > > >>> > > > >> wrote: > > > >> > > > >>> +1 for merge. > > > >>> > > > >>> Pipeline can exist in mixed version of DN (old and new). Design doc > > do > > > >> not > > > >>> provide much clarity about the behavior: > > > >>> - Weather write is supported in this case? allocating new blocks > and > > > >>> pipeline > > > >>> - How about pipeline / block already assigned for write? Client > will > > > not > > > >> be > > > >>> aware of changes. > > > >>> > > > >>> Since SCM is already finalied and DN finalization is in progress > > (which > > > >> can > > > >>> take 1-2 HB for few set of DNs where SCM will know). So there may > be > > a > > > >>> delay and DN upgrade may also take time. > > > >>> > > > >>> - How about DN supporting read and write operation when > finalization > > is > > > >> in > > > >>> progress? is it blocking or request will be rejected by DNs ? > > > >>> > > > >>> > > > >>> On Thu, Oct 1, 2026 at 7:46 PM Andrey Yarovoy via dev < > > > >>> [email protected]> > > > >>> wrote: > > > >>> > > > >>>> +1 for merge. > > > >>>> > > > >>>> On Thu, Oct 1, 2026 at 10:01 AM Uma Maheswara Rao Gangumalla < > > > >>>> [email protected]> wrote: > > > >>>> > > > >>>>> Great work. Thank you Ethan and all others for working on this. > > > >>>>> > > > >>>>> +1 for the merge. > > > >>>>> > > > >>>>> Regards, > > > >>>>> Uma > > > >>>>> > > > >>>>>> On Mon, Sep 28, 2026 at 4:02 PM Ethan Rose <[email protected]> > > > wrote: > > > >>>>>> > > > >>>>>>> Hi Ozone devs, > > > >>>>>>> > > > >>>>>>> This is a vote thread to merge the zero downtime upgrade > feature > > > >>>>> branch ( > > > >>>>>>> HDDS-14496-zdu) into master, which will enable zero downtime > > > >>> upgrades > > > >>>>>> from > > > >>>>>>> the first release containing this feature to all future > releases. > > > >>>>> Please > > > >>>>>>> familiarize yourself with the following documents when voting. > > All > > > >>> docs > > > >>>>> PRs > > > >>>>>> remain open so we can address questions and comments as they > arise > > > >>>> during > > > >>>>>> the voting process. > > > >>>>>> > > > >>>>>> - Design document <https://github.com/apache/ozone/pull/9664> > > > >>>>>> - Branch merge checklist < > > > >>>>> https://github.com/apache/ozone-site/pull/562 > > > >>>>>>> > > > >>>>>> - User Docs <https://github.com/apache/ozone-site/pull/545> > > > >>>>>> - Developer docs < > https://github.com/apache/ozone-site/pull/560 > > > >>> : > > > >>>>> This > > > >>>>>> guide covers how to handle compatibility for zero downtime > > > >> upgrade > > > >>>> and > > > >>>>>> should be understood by all developers and reviewers. > > > >>>>>> > > > >>>>>> Summary of changes: New Requirements for Developers > > > >>>>>> > > > >>>>>> *Once this branch is merged, all further commits must be ZDU > > > >>> compatible > > > >>>>> as > > > >>>>>> outlined in the developer guide*. > > > >>>>>> > > > >>>>>> The new versioning framework created on this branch provides > tools > > > >> to > > > >>>>>> safely incorporate incompatible changes, but it does not > > > >>> automatically > > > >>>>>> resolve them. That remains the responsibility of developers and > > > >>>>> reviewers. > > > >>>>>> Improved Developer Experience > > > >>>>>> > > > >>>>>> Each component now uses a single ComponentVersion to track all > > > >>>>> incompatible > > > >>>>>> changes across disk and network as outlined in the design doc. > > > >>>> Developers > > > >>>>>> no longer need to reason about whether their incompatible change > > > >>>>> requires a > > > >>>>>> LayoutFeature, ComponentVersion, or both. As part of this > change, > > > >> the > > > >>>>>> internal upgrade framework was rewritten and exposes a simpler > API > > > >> to > > > >>>>>> developers. This includes strongly typed component versions > (with > > > >>>> integer > > > >>>>>> conversion deferred until serialization), and an isSupportedBy > > > >> method > > > >>>> to > > > >>>>>> handle all version comparisons. > > > >>>>>> Improved Admin Experience > > > >>>>>> > > > >>>>>> To ensure finalization proceeds in the correct order as outlined > > in > > > >>> the > > > >>>>>> design document, admins no longer have to finalize OM and SCM > > > >>>>> separately. A > > > >>>>>> single ozone admin upgrade finalize command triggers > asynchronous > > > >>>>>> finalization throughout the cluster in the defined order. A > single > > > >>>> status > > > >>>>>> endpoint can be queried by ozone admin upgrade status, and > clients > > > >>> can > > > >>>>>> trigger and block on finalization with one ozone admin upgrade > > > >>> finalize > > > >>>>>> --wait command, which handles polling of the status endpoint by > > the > > > >>>>> client > > > >>>>>> automatically and is idempotent. > > > >>>>>> > > > >>>>>> Additionally, a Grafana dashboard has been added to provide a > > heads > > > >>> up > > > >>>>> view > > > >>>>>> of all components during an upgrade. It includes filtering to > zoom > > > >> in > > > >>>> on > > > >>>>> a > > > >>>>>> particular area and aggregates across Datanodes to handle large > > > >>>> clusters. > > > >>>>>> Removed Prepare For Upgrade > > > >>>>>> > > > >>>>>> The "prepare for upgrade" command which put OMs in a read-only > > mode > > > >>>>> before > > > >>>>>> an upgrade is no longer required. The CLI has been left as a > no-op > > > >>> for > > > >>>>>> compatibility with older upgrade scripts. See the developer > guide > > > >>>> linked > > > >>>>>> above for instructions to handle incompatible changes to OM > write > > > >>>>> requests. > > > >>>>>> Work In Progress > > > >>>>>> > > > >>>>>> There is some ongoing work we will continue in parallel with the > > > >>> merge > > > >>>>> vote > > > >>>>>> and finish before the branch is merged: > > > >>>>>> > > > >>>>>> - We are currently merging the latest master branch into the > > > >>> feature > > > >>>>>> branch, resolving conflicts, and running it through CI. The > > > >> final > > > >>>> hash > > > >>>>>> for > > > >>>>>> merge will be shared here when ready. > > > >>>>>> - The ZDU design doc will be updated based on the latest > copilot > > > >>>>> review > > > >>>>>> and other minor deviations identified from the resulting > > > >>>>> implementation. > > > >>>>>> - A PR to add a missed admin check on the ozone admin upgrade > > > >>> status > > > >>>>>> command is in flight: > > > >> https://github.com/apache/ozone/pull/11354 > > > >>>>>> > > > >>>>>> Future Work > > > >>>>>> > > > >>>>>> As mentioned in the merge checklist, the current OM request > > > >>> versioning > > > >>>>>> framework on master was left intact on the ZDU branch. However, > > the > > > >>>>> opt-in > > > >>>>>> annotation based approach does not suit the new ZDU requirements > > > >>> where > > > >>>>>> every new request needs to be versioned. Dev work has started > on a > > > >>> new > > > >>>>>> framework, but the change is large and was deliberately saved > for > > > >>>> master > > > >>>>>> after the branch merge so that it can be reviewed by a wider > > > >>> audience. > > > >>>>>> > > > >>>>>> Additionally, we will be investigating static analysis and AI > > > >> skills > > > >>> to > > > >>>>>> flag potentially incompatible changes during code reviews and > the > > > >>>> release > > > >>>>>> process. > > > >>>>>> ------------------------------ > > > >>>>>> > > > >>>>>> Thanks to Stephen, Zita, and Roland who also worked on the > > > >>> development > > > >>>> of > > > >>>>>> this feature and to everyone who shared inputs on the design. > > > >>>>>> > > > >>>>>> We will leave the vote thread open for at least 7 days, although > > > >> more > > > >>>>> time > > > >>>>>> may be required for developers to familiarize themselves with > the > > > >> new > > > >>>> ZDU > > > >>>>>> requirements. > > > >>>>>> > > > >>>>>> - Ethan > > > >>>>>> > > > >>>>> > > > >>>> > > > >>>> > > > >>>> -- > > > >>>> Thanks, > > > >>>> Andrey. > > > >>>> > > > >>> > > > >>> > > > >>> -- > > > >>> *Sumit Agrawal* | Senior Staff Engineer > > > >>> cloudera.com <https://www.cloudera.com> > > > >>> [image: Cloudera] <https://www.cloudera.com/> > > > >>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> > [image: > > > >>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > >> Cloudera > > > >>> on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > >>> ------------------------------ > > > >>> > > > >> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > >
