Re: [PR] HDDS-15138. SCM safemode pipeline rules should honor default EC replication config. [ozone]
rakeshadr commented on PR #10157: URL: https://github.com/apache/ozone/pull/10157#issuecomment-4648432155 Can you verify tests:- 1) Create RATIS file creation with replication factor 3. Both EC & RATIS should co-exists. 2) Restart SCM with EC files and make sure safemode check is out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HDDS-15138. SCM safemode pipeline rules should honor default EC replication config. [ozone]
rakeshadr commented on code in PR #10157:
URL: https://github.com/apache/ozone/pull/10157#discussion_r3371796166
##
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/HealthyPipelineSafeModeRule.java:
##
@@ -97,8 +101,7 @@ private int getMinHealthyPipelines(ConfigurationSource
config) {
HddsConfigKeys.HDDS_SCM_SAFEMODE_MIN_DATANODE,
HddsConfigKeys.HDDS_SCM_SAFEMODE_MIN_DATANODE_DEFAULT);
-// We only care about THREE replica pipeline
-return minDatanodes / HddsProtos.ReplicationFactor.THREE_VALUE;
+return minDatanodes / targetRequiredNodes;
Review Comment:
How abt to introduce new EC rule ?
```
SafeModeRuleFactory.java
SafeModeExitRule datanodeRule = new DataNodeSafeModeRule(eventQueue,
config, nodeManager, safeModeManager);
ECMinDataNodeSafeModeRule ecMinDnRule = new ECMinDataNodeSafeModeRule(
eventQueue, config, nodeManager, safeModeManager);
safeModeRules.add(ratisContainerRule);
safeModeRules.add(ecContainerRule);
safeModeRules.add(datanodeRule);
// Only adds a non-trivial gate when default replication is EC;
// the rule is a no-op (validate() == true) for RATIS-default clusters.
safeModeRules.add(ecMinDnRule);
```
```
ECMinDataNodeSafeModeRule.java
package org.apache.hadoop.hdds.scm.safemode;
import java.util.HashSet;
import java.util.Set;
import org.apache.hadoop.hdds.client.ECReplicationConfig;
import org.apache.hadoop.hdds.client.ReplicationConfig;
import org.apache.hadoop.hdds.conf.ConfigurationSource;
import org.apache.hadoop.hdds.protocol.DatanodeID;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
import org.apache.hadoop.hdds.scm.events.SCMEvents;
import org.apache.hadoop.hdds.scm.node.NodeManager;
import org.apache.hadoop.hdds.scm.node.NodeStatus;
import
org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer.NodeRegistrationContainerReport;
import org.apache.hadoop.hdds.server.events.EventQueue;
import org.apache.hadoop.hdds.server.events.TypedEvent;
/**
* Safemode exit rule for EC-default clusters.
*
* EC pipelines are ephemeral — they are created on-demand and do not
* survive SCM restarts. Therefore {@link HealthyPipelineSafeModeRule} cannot
* reliably gate safemode exit for EC. Instead, this rule ensures that at
least
* {@code data + parity} healthy DataNodes are registered before SCM exits
* safemode, guaranteeing that one full EC stripe can be written immediately
* after exit.
*
* The required node count is derived directly from the EC replication
* config (e.g. RS-3-2 → 5 nodes, RS-6-3 → 9 nodes), independent of
* {@code hdds.scm.safemode.min.datanode}.
*
* When the cluster default replication is not EC, this rule is a
* no-op ({@link #validate()} always returns {@code true}) and adds no
overhead.
*/
public class ECMinDataNodeSafeModeRule
extends SafeModeExitRule {
private final boolean enabled;
private final int requiredDns;
private final String ecConfigLabel;
private final NodeManager nodeManager;
private final Set registeredDnSet;
private int registeredDns = 0;
public ECMinDataNodeSafeModeRule(EventQueue eventQueue,
ConfigurationSource conf,
NodeManager nodeManager,
SCMSafeModeManager safeModeManager) {
super(safeModeManager, eventQueue);
this.nodeManager = nodeManager;
ReplicationConfig defaultConfig = getDefaultReplicationConfig(conf);
if (defaultConfig != null
&& defaultConfig.getReplicationType() ==
HddsProtos.ReplicationType.EC) {
ECReplicationConfig ecConfig = (ECReplicationConfig) defaultConfig;
this.requiredDns = ecConfig.getRequiredNodes(); // data + parity
this.ecConfigLabel = ecConfig.configFormat();
this.enabled = true;
this.registeredDnSet = new HashSet<>(requiredDns * 2);
SCMSafeModeManager.getLogger().info(
"ECMinDataNodeSafeModeRule enabled: EC default replication is {}, "
+ "requiring {} healthy DataNodes before safemode exit.",
ecConfigLabel, requiredDns);
} else {
this.requiredDns = 0;
this.ecConfigLabel = "";
this.enabled = false;
this.registeredDnSet = new HashSet<>(0);
SCMSafeModeManager.getLogger().debug(
"ECMinDataNodeSafeModeRule disabled: default replication is not
EC.");
}
}
@Override
protected TypedEvent getEventType() {
return SCMEvents.NODE_REGISTRATION_CONT_REPORT;
}
@Override
protected boolean validate() {
if (!enabled) {
return true;
}
if (validateBasedOnReportProcessing()) {
return registeredDns >= requiredDns;
}
return
Re: [PR] HDDS-15138. SCM safemode pipeline rules should honor default EC replication config. [ozone]
rakeshadr commented on code in PR #10157:
URL: https://github.com/apache/ozone/pull/10157#discussion_r3371742820
##
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/HealthyPipelineSafeModeRule.java:
##
@@ -97,8 +101,7 @@ private int getMinHealthyPipelines(ConfigurationSource
config) {
HddsConfigKeys.HDDS_SCM_SAFEMODE_MIN_DATANODE,
HddsConfigKeys.HDDS_SCM_SAFEMODE_MIN_DATANODE_DEFAULT);
-// We only care about THREE replica pipeline
-return minDatanodes / HddsProtos.ReplicationFactor.THREE_VALUE;
+return minDatanodes / targetRequiredNodes;
Review Comment:
EC safemode readiness isn't about pipelines — it's about having enough
healthy DNs registered
```
if (targetReplicationConfig.getReplicationType() == EC) {
// EC pipelines are ephemeral — they don't survive SCM restarts and
// are created on-demand. Requiring existing pipelines here would
// deadlock safemode on every restart. The DN safemode rule already
// enforces HDDS_SCM_SAFEMODE_MIN_DATANODE, so return 0 to make this
// rule a no-op for EC. The threshold floor of 0 means validate()
// passes if openPipelines >= floor(0 * percent) = 0.
return 0;
}
return minDatanodes / HddsProtos.ReplicationFactor.THREE_VALUE;
```
The DN count threshold belongs in a separate safemode exit rule — checking
that registeredHealthyDNs >= ecScheme.data + ecScheme.parity before exiting.
That's a clean addition to SCMSafeModeManager, not a hack into the pipeline
formula. Something like:
// In SCMSafeModeManager, when defaultReplication is EC:
int requiredDNs = defaultECConfig.getData() + defaultECConfig.getParity();
// validate: nodeManager.getNodeCount(NodeStatus.inServiceHealthy()) >=
requiredDNs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] HDDS-15138. SCM safemode pipeline rules should honor default EC replication config. [ozone]
aryangupta1998 commented on PR #10157: URL: https://github.com/apache/ozone/pull/10157#issuecomment-4432112262 @sodonnel, thanks, this is a good point. The pipeline safemode rules are there to ensure SCM has basic pipeline readiness before leaving safemode: **HealthyPipelineSafeModeRule** checks that enough selected open pipelines are healthy. **OneReplicaPipelineSafeModeRule** checks that selected open pipelines are reported by at least one DN. The issue was that selection was effectively hardcoded to `RATIS/THREE`. In an EC-default (or EC-only) setup, that can be wrong. A concrete case: with old behavior, the default threshold floor is 1 pipeline (`min.datanode=3 / 3`). If the cluster is EC-only and has no `RATIS/THREE` pipelines, this rule may never satisfy, and safemode can stay blocked unless config/workaround is used. This patch fixes that by making pipeline safemode checks follow `ReplicationConfig.getDefault(conf)`: If the default is EC, rules check EC pipelines; If the default is Ratis, behavior stays the same. I also updated BackgroundPipelineCreator so EC pipeline creation is included when the default is EC, so creation and safemode checks stay aligned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] HDDS-15138. SCM safemode pipeline rules should honor default EC replication config. [ozone]
sodonnel commented on PR #10157: URL: https://github.com/apache/ozone/pull/10157#issuecomment-4378964013 Its never really been clear to me what the purpose of the pipeline safemode rule is and how it differs from EC to Ratis. Could you explain why its important? Also, while a cluster can be configured to have EC as the default, that does not stop Ratis pipelines getting created and some data on the cluster being written in the Ratis format. The same goes the opposite way around too - a ratis default cluster can have EC data on it. One difference between EC and Ratis pipelines is that EC pipelines are short lived and Ratis pipelines are long lived. Ratis pipelines also survive cluster restarts, EC pipelines do not. Depending on the EC scheme, there will be a set of pipelines for each EC scheme 3-2, 6-3, 10-4 etc. I guess my main question is - if the cluster is default EC and only has EC data, what problem could occur if there is no safemode pipeline check? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
