[
https://issues.apache.org/jira/browse/HDDS-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137222#comment-18137222
]
Stephen O'Donnell commented on HDDS-17526:
------------------------------------------
> Register a post finalise action which starts a thread which begins closing
> the legacy pipelines matching certain conditions using the existing close and
> delete pipeline lifecycle.
[~anuragparvatikar] This may be something that is better built into the
pipeline scrubber / creator added by the new feature, where it checks the
apparent version in use and adjusts its code path on each pass based on its
value. Just keep in mind that upgrade actions are generally intended to be fast
operations - not something that should block the finalization call for a long
period of time.
In this case, this feels like a generic problem you need to solve for this
feature irrespective of ZDU. Even with a cold start at the new version, all the
old pipelines are loaded from disk and the cluster will end up with a full
quota of pipelines when it starts. Then the storage policy changes will have to
deal with that somehow.
> ZDU Impact on Storage Policy Feature [HDDS-11233]
> -------------------------------------------------
>
> Key: HDDS-17526
> URL: https://issues.apache.org/jira/browse/HDDS-17526
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Datanode, Ozone Filesystem, Ozone Manager, Ozone
> Recon
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
> Attachments: Storage_Policy_ZDU_Design_with_Jira_Subtasks_v3.md
>
>
> The Storage Policy feature in HDDS-11233 is under development and started
> before the upstream zero-downtime upgrade (ZDU) framework was merged in
> master. The feature branch currently declares
> BUCKET_STORAGE_POLICY_SUPPORT(12) in the legacy OMLayoutFeature enum, and
> some OM request validators use that legacy value. After incorporating
> upstream ZDU, migrate this feature to the new component-version framework and
> establish explicit, testable compatibility boundaries for all components
> participating in policy metadata handling and physical data placement.
> OM must not persist or activate unsupported storage-policy metadata before
> the appropriate finalization. Clients and S3 Gateway must negotiate the
> capability of the receiving OM and preserve legacy behavior. SCM must
> validate policy-aware allocation, recovery and move capabilities
> independently of OM finalization; DataNodes must not silently ignore or
> misapply explicit storage-type requests. RATIS/EC repair, Container Balancer
> and Disk Balancer must maintain the documented strict/fallback placement
> contract. Recon and operational tooling must remain compatible with
> mixed-version metadata and report actionable failures.
> Deliver the implementation changes, version/field compatibility contract, and
> mixed-version unit/integration tests. Do not assume that ZDU finalization,
> Protobuf field acceptance, or RPC success proves actual physical placement
> compliance. Verify exact source paths and version numbers against the merged
> HDDS-11233 working tree.
> *Acceptance criteria:*
> * Storage Policy is registered through the correct post-ZDU OM
> component-version sequence, with an explicit decision for any persisted
> legacy feature-branch layout 12 state.
> * Pre-finalization OM operations preserve legacy behavior and do not persist
> new incompatible policy state; explicit-policy request handling is agreed and
> tested.
> * Client/S3G and each server boundary use appropriate negotiated capability
> checks and authoritative server-side enforcement.
> * Strict and allowed-fallback semantics are consistent across allocation,
> RATIS/EC repair, balancing and actual DataNode volume selection.
> * Mixed-version upgrade, restart, replay, snapshot, HA failover and
> downgrade-before-finalization scenarios pass, or have explicitly documented
> unsupported states.
> * Recon and operational signals remain usable throughout rollout;
> implementation and tests are linked to the corresponding tickets.
> cc: [~umamaheswararao] [~kkasawa] [~anuragparvatikar]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
