[
https://issues.apache.org/jira/browse/HDDS-17059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devesh Kumar Singh updated HDDS-17059:
--------------------------------------
Summary: Introduce a generic Jobworker framework for OM-managed background
jobs. (was: Implement Jobworker-based key migration to satisfy storage
policies)
> Introduce a generic Jobworker framework for OM-managed background jobs.
> -----------------------------------------------------------------------
>
> Key: HDDS-17059
> URL: https://issues.apache.org/jira/browse/HDDS-17059
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode, Ozone Filesystem, Ozone Manager
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
>
> Updating a bucket’s storage policy changes the desired placement for
> applicable writes, but existing keys remain on their current storage volumes.
> For example, changing a bucket from HOT to COLD does not move existing RATIS
> THREE keys from SSD to ARCHIVE.
> Introduce asynchronous key migration using Jobworker to rewrite existing keys
> onto the target storage tier while preserving their replication configuration.
> Bucket-policy updates through CLI or native APIs and migration submission are
> separate operations; updating the policy alone must not be represented as
> completed data migration.
> *Acceptance criteria:*
> - A RATIS THREE key can migrate from SSD to ARCHIVE while retaining its name,
> content, replication configuration, and required attributes.
> - Placement respects the target policy and configured fallback rules.
> - Failed or incomplete copies cannot replace the committed source key.
> - Concurrent overwrites or deletions cannot be overwritten by stale migration
> work.
> - Tasks recover across OM failover and Jobworker restart without losing work
> or double-counting results.
> - Progress, failures, retries, and cancellation outcomes are observable.
> - Integration tests verify actual DN placement, data correctness, failure
> recovery, and unchanged replication.
> The migration flow should:
> 1. Persist migration tasks and batches of key names, observed generations,
> and target storage policies in OM.
> 2. Allow lifecycle processing to create migration work.
> 3. Use StoragePolicySatisfierService to schedule tasks, select healthy
> Jobworkers, and dispatch commands through heartbeat responses.
> 4. Have Jobworker read each source key and invoke policy-aware rewriteKey,
> retaining the source replication configuration.
> 5. Reuse the OM → SCM → datanode write path to allocate and write target-tier
> blocks.
> 6. Validate the source generation before committing the replacement,
> protecting concurrent user updates.
> 7. Report per-key outcomes, retry eligible failures, and persist task
> progress through OM Ratis.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
