spacemonkd opened a new pull request, #10683: URL: https://github.com/apache/ozone/pull/10683
## What changes were proposed in this pull request? HDDS-11900. Introduce fine-grained locking for OBS key operations to enable leader-execution model. ## Description This pull request introduces fine-grained key-path-level locking for Ozone Object Store (OBS) key operations as a prerequisite for the leader-execution model (HDDS-11898). Currently, all OBS key operations within a bucket are serialized by a single bucket-level write lock (`BUCKET_LOCK`), preventing concurrent operations on different keys. This change enables parallel key operations by introducing a lock hierarchy that allows bucket-level read locks (preventing bucket deletion) paired with per-key write locks. ### Problem Statement The leader-execution design requires deterministic, repeated execution of requests: leaders execute request logic and produce a side-effect log, while followers replay the log to apply only the side effects (DB mutations). This model breaks if concurrent mutations on different keys within a bucket are possible—replaying in an arbitrary order risks different outcomes than the leader's execution. Today's coarse bucket-level write locks serialize all key operations, making determinism straightforward but limiting throughput. The solution is to introduce fine-grained locking: keys within a bucket can be locked independently using striped locks, enabling concurrent operations on different keys while maintaining determinism through ordered multi-key lock acquisition (`bulkGet()`). ### Changes Proposed **Lock Infrastructure:** - Extended `OzoneLockStrategy` interface with default methods for multi-key locks and bucket-read-only locks, enabling backward-compatible migration of handlers. - Implemented `OBSKeyPathLockStrategy`: acquires BUCKET_LOCK read + per-key KEY_PATH_LOCK writes using ordered `bulkGet()` for multi-key operations; bucket-read-only operations acquire only BUCKET_LOCK read. - Implemented `RegularBucketLockStrategy` fallback: preserves BUCKET_LOCK write behavior when key-path locking is disabled (config default: `ozone.om.key.path.lock.enabled=false`). **Request Handler Migrations:** - Single-key write operations: OMKeyCommitRequest, OMKeyDeleteRequest, OMAllocateBlockRequest, OMKeySetTimesRequest, S3MultipartUploadCompleteRequest, S3MultipartUploadAbortRequest, S3MultipartUploadCommitPartRequest, S3DeleteObjectTaggingRequest, S3PutObjectTaggingRequest, OMKeyAclRequest. - Bucket-read-only operations (no key lock): OMKeyCreateRequest, S3InitiateMultipartUploadRequest (per design doc, these write only to OpenKey table with unique clientIDs; no key conflict). - Multi-key ordered locks: OMKeysDeleteRequest (locks all deleted keys), OMKeyRenameRequest (locks source + destination), OMKeysRenameRequest (locks all from/to keys). ### Design & Approach The implementation follows the OBS locking strategy defined in : - **Lock Hierarchy**: S3_BUCKET_LOCK(0) → VOLUME_LOCK(1) → BUCKET_LOCK(2) → KEY_PATH_LOCK(5) - **Bucket-read + key-write model**: All key operations acquire BUCKET_LOCK read (prevents bucket deletion) + KEY_PATH_LOCK write (per-key serialization). - **Ordered multi-key locking**: Batch operations like DeleteKeys and Rename use `bulkGet()`-based ordered lock acquisition to guarantee deadlock-freedom. - **Backward compatibility**: Default config keeps key-path locking disabled; RegularBucketLockStrategy falls back to BUCKET_LOCK write, preserving existing behavior until the feature is enabled. The migration uses the Strategy pattern (`OzoneLockStrategy`) to avoid coupling handlers to specific lock schemes, allowing per-bucket-layout lock selection (OBS vs FSO) and per-config feature gating. ### References: @kerneltime's reference specification: https://github.com/kerneltime/ozone/tree/9c59ae0d66227ce10849b4f914f9efdce9318103/hadoop-hdds/docs/content/design/leader-execution Pre-Ratis Execution design document: https://github.com/apache/ozone/pull/10503 ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-11900 ## How was this patch tested? Unit Tests: - New `TestOBSKeyPathLockStrategy`: single-key, multi-key, bucket-read-only lock operations; concurrent lock acquisition on different keys; deadlock-free ordered locking. - Updated existing tests: fixed mock setup in `TestOMKeyRequest` and `TestS3MultipartRequest` base classes to mock `getOzoneLockProvider()`. Integration Tests (all passing) Compilation: `mvn clean install -U -DskipTests -DskipShade` **Feature Flag:** - Default config: `ozone.om.key.path.lock.enabled=false` (no behavior change; uses RegularBucketLockStrategy). - When enabled: OBSKeyPathLockStrategy activates per-key locking for OBS buckets, enabling concurrent key operations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
