hani-fouladgar opened a new pull request, #11571: URL: https://github.com/apache/ozone/pull/11571
## What changes were proposed in this pull request? Apache Ozone's Ozone Manager (OM) runs a background `KeyDeletingService` that sends deleted key blocks to SCM via `ScmBlockLocationProtocol.deleteKeyBlocks`. On SCM, each call writes transactions into the Ratis-replicated DeletedBlockLog. When SCM's Ratis followers fall behind the leader, this asynchronous deletion load is exactly what backs up on SCM's replication, yet OM has no way to detect the condition — it only self-limits with static knobs (`keyLimitPerTask`, Ratis byte batching), which do not react to SCM's state. This PR adds a feedback loop so OM throttles its block-deletion load to SCM when SCM followers are highly lagged: - The SCM leader computes its follower replication lag — `leaderLastAppliedIndex − min(followerNextIndices)`, expressed in Raft log entries — using the same RaftServer.Division APIs already used elsewhere in the codebase (e.g. the datanode ContainerStateMachine and OM's allowFollowerReadLocalLease). - SCM advertises that lag as an advisory field on the existing `deleteKeyBlocks` response. The field is an additive `optional` field on `DeleteScmKeyBlocksResponseProto`. - OM records the reported lag and, on subsequent `KeyDeletingService` runs, reduces (does not stop) its per-task key deletion limit while the lag is at or above a configurable threshold. As the lag clears, OM automatically returns to the full limit. Why the signal is carried on the success response rather than as a server-side rejection: returning the lag on a normal, successful response means the in-flight deletion batch still commits — OM wastes no `deletedTable` scan work and never trips the SCM failover/retry path (`SCMFailoverProxyProviderBase`), which on an error would retry or fail over to another SCM and amplify load on an already-struggling SCM. Reducing rather than stopping the load keeps OM sending smaller batches, so it keeps receiving fresh lag readings and can un-throttle — avoiding a stuck state. The scope is intentionally limited to block deletion (the true async/background load); block allocation (the foreground client write path) is untouched. Measurement lives on SCM (which alone can observe its follower lag); the throttle policy lives on OM (where the deletion knobs already are). The feature is enabled by default with a deliberately conservative threshold, so only genuinely severe lag triggers throttling; healthy clusters are unaffected. New configuration (OM): - `ozone.om.key.deleting.scm.follower.lag.throttle.threshold` = 100000; Follower lag (Raft log entries) at/above which OM throttles. <=0 disables. - `ozone.om.key.deleting.scm.lag.throttled.limit.per.task` = 1000 ; Reduced per-task key limit used while throttled (must be >0). Compatibility: the new proto field is additive and optional, so mixed-version clusters are safe — an old SCM reports no signal (field absent → 0 → no throttle) and an old OM simply ignores the field. This is an RPC-contract change (SCM block protocol) and is called out here for review. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17075 ## How was this patch tested? Unit tests (new/extended): - `TestSCMRatisServerImpl#testGetFollowerReplicationLag` — verifies the leader-side lag computation: lag = `leaderIndex − min(followerNextIndices`); returns 0 when not the leader, when there are no followers (non-HA), and clamps negatives to 0. - `TestKeyDeletingService` (nested `Normal`) — verifies the service reduces its per-task limit to the throttled value and increments the throttle metric when the reported lag is at/above the threshold, uses the full limit below the threshold, and that a non-positive threshold disables throttling entirely. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
