[
https://issues.apache.org/jira/browse/HDDS-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18134638#comment-18134638
]
Ivan Andika edited comment on HDDS-11693 at 10/30/26, 3:25 AM:
---------------------------------------------------------------
[~priyeshkaratha] Seems [~nanda] is not actively working on this so I'll take a
stab at it.
> However, currently the {{SafeModeExitRule}} triggers its validation logic
> upon receiving datanode reports. If we are planning to remove the processing
> logic from the safe mode rule validations, I would like to understand what
> the new trigger point for these validations will be.
My guess is that the incremental method of calling the SafeMode#process event
handler to
handle NodeRegistrationContainerReport event triggered a lot of issues and
therefore it is going to be replaced by simpler way just checking the state
periodically in validate. So SafeModeManager just check the state of the SCM at
every refresh interval. Currently, we are still setting
validateBasedOnReportProcessing to true for every case, but in the future, we
either make this configurable or just remove it entirely.
> Is there any existing design documentation related to this refactoring? Also,
> has there been any prior design discussion or background context around this
> change that I can refer to?
I don't think there is one. The point is that the incremental method causes
some edge cases.
For example, in AbstractContainerSafeModeRule, we initialize Map<ContainerID,
Integer> containers and totalContainers once. However, these are unnecessary
duplicated states and instead of making ContainerManager /
ContainerStateManager / ContainerStateMap be the source of truth we have to
keep track of these two states that can diverge easily.
I suspect these triggered a lot of edge cases that makes SafeMode unstable. I
guess the initial intent is that to prevent unnecessary ContainerManager copy
although we might be able to think on the solution of preventing this container
copy.
So I guess we just follow the TODO in HDDS-11958 by removing the report based
processing
{code:java}
// TODO: Report processing logic will be removed in future. HDDS-11958.
// This flag is to add new code without breaking Safemode logic until we have
HDDS-11958.
private boolean validateBasedOnReportProcessing = true;
{code}
was (Author: JIRAUSER298977):
[~priyeshkaratha] Seems [~nanda] is not actively working on this so I'll take a
stab at it.
> However, currently the {{SafeModeExitRule}} triggers its validation logic
> upon receiving datanode reports. If we are planning to remove the processing
> logic from the safe mode rule validations, I would like to understand what
> the new trigger point for these validations will be.
My guess is that the incremental method of calling the SafeMode#process event
handler to add the
handle NodeRegistrationContainerReport event is going to be replaced by just
checking the state periodically in validate. So SafeModeManager just check the
state of the SCM at every refresh interval. Currently, we are still setting
validateBasedOnReportProcessing to true for every case, but in the future, we
either make this configurable or just remove it entirely.
> Is there any existing design documentation related to this refactoring? Also,
> has there been any prior design discussion or background context around this
> change that I can refer to?
I don't think there is one. The point is that the incremental method causes
some edge cases. For example, in AbstractContainerSafeModeRule, we initialize
Map<ContainerID, Integer> containers and totalContainers. However, these are
duplicated states and instead of making ContainerManager /
ContainerStateManager / ContainerStateMap be the source of truth we have to
keep track of these two states that can diverge easily. I suspect these
triggered a lot of edge cases.
So I guess we just follow the TODO in HDDS-11958 by removing the report based
processing
{code:java}
// TODO: Report processing logic will be removed in future. HDDS-11958.
// This flag is to add new code without breaking Safemode logic until we have
HDDS-11958.
private boolean validateBasedOnReportProcessing = true;
{code}
> SCM Safemode Improvements
> -------------------------
>
> Key: HDDS-11693
> URL: https://issues.apache.org/jira/browse/HDDS-11693
> Project: Apache Ozone
> Issue Type: Improvement
> Components: SCM
> Reporter: Nandakumar
> Assignee: Nandakumar
> Priority: Major
>
> This parent Jira tracks the SCM Safemode improvements.
> Currently, the Safemode Rule validation is driven by processing of reports
> sent by Datanodes, this is unnecessary and this also introduces additional
> complexity as every rule is maintaining the same information that the
> NodeManager, PipelineManager & the ContainerManager are maintaining.
> Since the information about Datanodes/Pipelines/Containers are maintained at
> two different places, this can go out of sync and introduce bugs into
> safemode logic. (eg: HDDS-5263)
> We will move to a simpler model where the SafemodeManager will not care about
> report processing (Safemode Rules will not process any Datanode Reports) but
> check NodeManager, PipelineManager and ContainerManager to validate the exit
> rule.
> With this change the Safemode logic will become simple and easy to follow.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
