hani-fouladgar commented on PR #11571: URL: https://github.com/apache/ozone/pull/11571#issuecomment-6345195736
@ashishkumar50 Q1 (OM DB bloat): Thanks, great catch — this is an important tradeoff to call out. You're right that it's intentional: the backlog has to live somewhere during SCM lag, and the alternative is pushing it into SCM's already-struggling Ratis `DeletedBlockLog`, which worsens the lag and still doesn't let the deletes commit. Throttling keeps that backlog local (plain RocksDB metadata, not the replicated path) and self-drains once lag clears — note it _reduces_, not _stops_, so OM keeps sending at whatever rate SCM can absorb, and the threshold (100k entries) only trips on genuinely severe lag. You're right that an indefinitely broken SCM means unbounded growth, but no design completes deletes while SCM can't replicate them. Q2 (why only deletion): Good question, thanks. Deletion is the only OM→SCM load that's both bulk-async and safe to slow. `deleteKeyBlocks` writes to the Ratis-replicated `DeletedBlockLog` and can burst to millions of entries with no user waiting — ideal to throttle. The other Ratis-touching op is block allocation, which is (a) the foreground client-write path (throttling it degrades user I/O — a different policy), and (b) backed by a batched `SequenceIdGenerator` that only persists to Ratis once per ID range, so far lighter. Everything else (getScmInfo, sortDatanodes, container lookups) is reads, no Ratis sync. Backpressuring the foreground path belongs with the OM handler-pool work (HDDS-16596), so it's intentionally out of scope here. Q3 (auto bootstrap): Thanks — good point to raise. It actually already does: a far-behind follower is caught up via Ratis InstallSnapshot (`SCMStateMachine` + `SCMSnapshotProvider`), independent of this PR. But bootstrap and throttle are complementary, not alternatives: bootstrap recovers a follower that already fell off (heavy, last resort); this throttle reduces the write rate during moderate lag so it doesn't escalate to that point, and lets the follower catch up without a snapshot install. Bootstrap alone wouldn't help — it doesn't reduce write pressure, so a follower overwhelmed by volume would just re-lag. The lag signal here could later also drive an SCM-side snapshot trigger, but that's SCM-internal and separate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
