Devesh Kumar Singh created HDDS-17073:
-----------------------------------------
Summary: Implement Jobworker-based key migration to satisfy
storage policies
Key: HDDS-17073
URL: https://issues.apache.org/jira/browse/HDDS-17073
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Datanode, Ozone Filesystem, Ozone Manager
Reporter: Devesh Kumar Singh
Assignee: Devesh Kumar Singh
Existing keys retain their current physical placement when a bucket’s storage
policy changes. For example, changing a bucket from HOT to COLD does not move
an existing RATIS THREE key from SSD to ARCHIVE.
Use the Jobworker framework to asynchronously rewrite selected existing keys
onto the target storage tier while preserving their replication configuration.
A bucket-policy update records placement intent. Migration requires a separate
submission or producer, such as lifecycle processing, to identify eligible keys
and create migration work.
*Scope:*
- Persist migration tasks and transaction batches in OM.
- Record each key’s observed generation and the target storage policy.
- Connect lifecycle transition processing to migration-task creation.
- Introduce StoragePolicySatisfierService to schedule pending work, limit
concurrency, and select healthy Jobworkers.
- Add a migration command handler that reads source data and invokes
policy-aware rewriteKey with the source replication configuration unchanged.
- Reuse OM → SCM → datanode allocation and write handling to enforce target
placement and permitted fallback.
- Preserve required key attributes and validate copied data.
- Process per-key results, retry eligible failures, and maintain durable
progress through OM Ratis.
- Support cancellation, failure cleanup, and recovery after OM failover or
worker loss.
*Acceptance criteria:*
- A RATIS THREE key can move from SSD to ARCHIVE while retaining its logical
name, content, size, replication configuration, and required attributes.
- Tests verify actual DN volume placement alongside OM and SCM metadata.
- Failed or incomplete copies cannot replace the committed source key.
- Concurrent user overwrites or deletions prevent stale migration commits.
- Retries, duplicate results, and failover do not lose work or double-count
outcomes.
- Cancellation stops remaining scheduling work and clearly reports that
already-committed migrations are not rolled back.
- Source and abandoned-target cleanup respects applicable snapshot and
retention semantics.
- Unsupported target placement produces a clear failure or explicitly permitted
fallback.
- Progress and failures are observable through task status, counters, and
diagnostics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
