[
https://issues.apache.org/jira/browse/HDDS-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269801#comment-17269801
]
Baolong Mao edited comment on HDDS-4656 at 1/22/21, 5:32 AM:
-------------------------------------------------------------
[~linyiqun] [~kennethlnnn] [~ljain] Thanks for your review and reply, I'm glad
you are interested in this proposal.
> `balancer rate limit is also very important`
Yeah, definitely, will finish that part.
> I think it might also be good to look at pros and cons of having a separate
> service.
Yeah, there would be lots of pros if we put the balancer into the SCM.
- It would be easier to use and sync with ReplicationManager for balancing the
containers.
- If needed the balancing commands can be provided as part of admin command
line.
- It can use the rack aware placement policy in SCM to provide default
balancing feature.
But it have some cons too.
- It would affect the SCM core logic
- Hard to implement and update the balancer related code.
- Make SCM fat if we put lots features into it while some of these features can
be an independent tool.
> let container balancer as an independent program tool like HDFS balancer +1
Thanks for your +1, hope you can try this feature while we finish it.
was (Author: maobaolong):
[~linyiqun] [~kennethlnnn] [~ljain] Thanks for your review and reply, I'm glad
you are interested in this proposal.
> `balancer rate limit is also very important`
Yeah, definitely, will finish that part.
> I think it might also be good to look at pros and cons of having a separate
> service.
Yeah, there would be lots of pros if we put the balancer into the SCM.
- It would be easier to use and sync with ReplicationManager for balancing the
containers.
- If needed the balancing commands can be provided as part of admin command
line.
- It can use the rack aware placement policy in SCM to provide default
balancing feature.
But it have some cons too.
- It would affect the SCM core logic
- Hard to implement and update the balancer related code.
- Make SCM fat if we put lots features into it while some of these features can
be an independent tool.
> Add a container balancer tool or service for HDDS
> -------------------------------------------------
>
> Key: HDDS-4656
> URL: https://issues.apache.org/jira/browse/HDDS-4656
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
> Components: Ozone Datanode, SCM, Tools
> Affects Versions: 1.1.0
> Reporter: Baolong Mao
> Assignee: Baolong Mao
> Priority: Major
>
> When an existing Ozone cluster is nearly full, we have to add more datanodes
> into the Ozone cluster, but there are two issue we must face.
> - When new allocate container request coming, SCM should better to choose the
> datanodes in low usage, if not, the performance will getting pool.
> - For read request, the existing datanodes stored lots of blocks, so they are
> responsible for serving the read request and supply the data stream service,
> meanwhile, the new coming datanodes can help nothing.
> If we have a balancer tool just like hdfs balancer, we can move the block or
> container from some high usage datanodes to low, I think this is one of
> necessary tools for Ozone.
> container balancer design doc
> https://docs.google.com/document/d/15PdYaP6aLB18ptbcOK3XWlL4Y1PLfIP-ll30KjNPZ2g/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
