sarvekshayr opened a new pull request, #11409: URL: https://github.com/apache/ozone/pull/11409
## What changes were proposed in this pull request? When an SCM follower restarts after being down for a long time, Ratis syncs the log to disk quickly, but `applyTransaction` lags far behind, so `commitIndex` can exceed `appliedIndex` by millions of entries. All the DB updates produced while applying that backlog accumulate in the single in-memory `BatchOperation` held by `SCMHADBTransactionBufferImpl`, causing heavy memory pressure. This change adds a count-based flush on the apply path: - New config `ozone.scm.ha.dbtransactionbuffer.flush.pending.limit` (default `10000`): the number of buffered DB operations after which the buffer is flushed. Values `<= 0` are invalid and fall back to the default with a warning. - New `SCMHADBTransactionBuffer#flushIfPendingLimitReached()` - `SCMStateMachine#applyTransaction` calls it right after `updateLatestTrxInfo(...)` and `updateLastAppliedTermIndex(...)`, still inside the `beginApplyingTransaction()`/`endApplyingTransaction()` window. Flushing after the index is recorded keeps the persisted `TransactionInfo` consistent with the committed batch. The existing interval-based flush is unchanged. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-16666 ## How was this patch tested? Added unit tests in `TestSCMHADBTransactionBufferImpl` and `TestSCMStateMachine`. Manually tested on a SCM HA docker cluster with the flush pending limit set to 5: - Flush lines were observed on all three SCMs, including followers, whenever the buffered operation count reached the limit. (Note: Changed log level to `INFO` for testing purpose) - With one follower stopped, load was generated, and the follower restarted: it logged repeated flushes while replaying the backlog and then `Follower caught up with leader`. ``` scm3-1 | 2026-10-05 06:43:38,126 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing scm3-1 | 2026-10-05 06:43:38,130 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing scm3-1 | 2026-10-05 06:43:38,133 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing scm3-1 | 2026-10-05 06:43:38,138 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing scm3-1 | 2026-10-05 06:43:38,141 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
