runzhiwang commented on a change in pull request #1371:
URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r492416925
##########
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##########
@@ -98,8 +105,65 @@ private boolean exceedPipelineNumberLimit(ReplicationFactor
factor) {
return false;
}
+ private Map<DatanodeDetails, Integer> getSuggestedLeaderCount(
+ List<DatanodeDetails> dns) {
+ Map<DatanodeDetails, Integer> suggestedLeaderCount = new HashMap<>();
+ for (DatanodeDetails dn : dns) {
+ suggestedLeaderCount.put(dn, 0);
+
+ Set<PipelineID> pipelineIDSet = getNodeManager().getPipelines(dn);
+ for (PipelineID pipelineID : pipelineIDSet) {
+ try {
+ Pipeline pipeline =
getPipelineStateManager().getPipeline(pipelineID);
+ if (!pipeline.isClosed()
+ && dn.getUuid().equals(pipeline.getSuggestedLeaderId())) {
Review comment:
> Plan weight for each of node as a leader when the cluster has
thousands of nodes can be difficult.
If each node has similar hardware, i.e. CPU, memory, we just plan weight as
now, assign each node with same leader number, it is cheap and reasonable.
I think the only case we need to consider is that some nodes' hardware is
weaker than other nodes' obviously. I think the weeker datanodes should engage
in less pipeline than the stronger datanodes, but ozone does not support this
now. If we can support this, the maxum leader number of each datanode should be
less or equal to ((1/3) * the pipeline number it engaged in), and we select
the datanode as the leader which has lowest value of (leader number / pipeline
number it engaged in) in 3 datanodes, this is also cheap.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
