errose28 opened a new pull request, #12024:
URL: https://github.com/apache/ozone/pull/12024

   **DRAFT another proof-of-concept solution to HDDS-17580 which would replace 
#11956 and #11861**
   
   ## What changes were proposed in this pull request?
   
   ### Implementation
   
   SCM does not persist datanode registrations, but it does persist pipelines, 
which are reloaded `OPEN` on restart. If a pipeline member never re-registers 
(for example, it died while SCM was down), the pipeline can still be used for 
writes and is returned for reads of its open containers. This lasts until the 
pipeline scrubber closes it, which only began 5 minutes after safemode exit and 
then ran every 150s. EC pipelines are hit hardest because datanodes never close 
them on their own.
   
   This change makes sure such pipelines are closed before SCM leaves safemode:
   
   - **Safemode rule:** `HealthyPipelineSafeModeRule` cannot pass while any 
`OPEN` pipeline, of any replication type, has a member that is not registered 
with SCM. The rule is now always registered. Its existing RATIS/THREE 
healthy-pipeline threshold is still enabled or disabled under the same 
conditions as before.
   - **Pipeline scrubber:** the scrubber is now its own `SCMService`, 
`BackgroundPipelineScrubber`, similar to `BackgroundPipelineCreator`. On the 
leader it also runs during safemode, once:
     - the container safemode rules have passed (signalled by a new 
`CONTAINER_SAFE_MODE_RULES_VALIDATED` event), and
     - `ozone.scm.stale.node.interval` has elapsed since SCM startup. A 
datanode still unregistered at that point is handled like a stale node, whose 
pipelines `StaleNodeHandler` already closes.
   
     Outside safemode it runs without the previous post-exit wait. 
`hdds.scm.wait.time.after.safemode.exit` still applies to the other services 
that use it.
   - **Scrubbing:** a failure on one pipeline no longer aborts the rest of the 
pass, except for not-leader errors. When a pass closes any pipeline, it 
triggers a one-shot `BackgroundPipelineCreator` run so replacement RATIS 
pipelines are created right away. This also works when 
`hdds.scm.safemode.pipeline.creation=false`. EC pipelines are still created on 
demand.
   
   ### DRAFT Pending Further Verification:
   
   - Behavior change: if a datanode stays down across an SCM restart, safemode 
now lasts at least one stale node interval (5 minutes by default) after 
startup. If every datanode registers again, safemode exit is not delayed.
   
   - Known limitation, not changed here: `OneReplicaPipelineSafeModeRule` 
rebuilds its denominator from the pipelines that are currently `OPEN`. Once the 
scrubber closes pipelines whose members are all unregistered, the rule passes 
even though the open containers on those pipelines have no readable replica.
   
   - Changes to `BackgroundPipelineScrubber` might be better split to their own 
PR since they what makes this diff so large:
       - `BackgroundPipelineScrubber` was split to its own class implementing 
`SCMService` instead of `BackgroundSCMService`. This is due to the unique event 
and timing triggers it now has. This is similar to `BackgroundPipelineCreator`, 
but it creates a lot of boilerplate which should be combined. 
       - A TODO marks moving the scrub logic out of `PipelineManagerImpl` and 
into `BackgroundPipelineScrubber`, which would create a larger diff.
   
   ## What is the link to the Apache JIRA
   
   HDDS-17580
   
   ## How was this patch tested?
   
   - New unit tests:
     - `TestBackgroundPipelineScrubber`: safemode gating, startup delay, 
follower-to-leader handoff, and waking up without waiting for the interval.
     - `TestPipelineManagerImpl`: scrubbing continues after one pipeline fails; 
the creator is triggered only when a pass closes pipelines.
     - `TestHealthyPipelineSafeModeRule`: open RATIS/EC pipelines with 
unregistered datanodes block the rule.
     - `TestSCMSafeModeManager`: `CONTAINER_SAFE_MODE_RULES_VALIDATED` is fired 
once.
     - `TestBackgroundPipelineCreator`: the scrubber event triggers a one-shot 
run in safemode.
   - Updated `TestSafeModeRuleFactory` and the EC-default rule test in 
`TestSCMSafeModeManager`.
   - Updated the integration test 
`TestSCMSafeModeWithPipelineRules#testScmSafeMode`: it restarts SCM with two 
datanodes still down and checks that SCM stays in safemode until their 
pipelines are closed, and that no open pipeline has an unregistered datanode at 
exit.
   
   🤖 Generated with [[Claude 
Code](https://claude.com/claude-code)](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to