Devesh Kumar Singh created HDDS-17059:
-----------------------------------------
Summary: Implement Jobworker-based key migration to satisfy
storage policies
Key: HDDS-17059
URL: https://issues.apache.org/jira/browse/HDDS-17059
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Datanode, Ozone Filesystem, Ozone Manager
Reporter: Devesh Kumar Singh
Assignee: Devesh Kumar Singh
Updating a bucket’s storage policy changes the desired placement for applicable
writes, but existing keys remain on their current storage volumes. For example,
changing a bucket from HOT to COLD does not move existing RATIS THREE keys from
SSD to ARCHIVE.
Introduce asynchronous key migration using Jobworker to rewrite existing keys
onto the target storage tier while preserving their replication configuration.
Bucket-policy updates through CLI or native APIs and migration submission are
separate operations; updating the policy alone must not be represented as
completed data migration.
*Acceptance criteria:*
- A RATIS THREE key can migrate from SSD to ARCHIVE while retaining its name,
content, replication configuration, and required attributes.
- Placement respects the target policy and configured fallback rules.
- Failed or incomplete copies cannot replace the committed source key.
- Concurrent overwrites or deletions cannot be overwritten by stale migration
work.
- Tasks recover across OM failover and Jobworker restart without losing work or
double-counting results.
- Progress, failures, retries, and cancellation outcomes are observable.
- Integration tests verify actual DN placement, data correctness, failure
recovery, and unchanged replication.
The migration flow should:
1. Persist migration tasks and batches of key names, observed generations, and
target storage policies in OM.
2. Allow lifecycle processing to create migration work.
3. Use StoragePolicySatisfierService to schedule tasks, select healthy
Jobworkers, and dispatch commands through heartbeat responses.
4. Have Jobworker read each source key and invoke policy-aware rewriteKey,
retaining the source replication configuration.
5. Reuse the OM → SCM → datanode write path to allocate and write target-tier
blocks.
6. Validate the source generation before committing the replacement, protecting
concurrent user updates.
7. Report per-key outcomes, retry eligible failures, and persist task progress
through OM Ratis.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
