[
https://issues.apache.org/jira/browse/HDDS-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreeja updated HDDS-16179:
--------------------------
Description:
The assessment command in HDDS-16174 tells users how much cluster imbalance
there is. The next step is estimating how long balancing will take and how many
iterations it will need — without actually moving containers or calling SCM
planning logic.
Introduce three profiles slow / medium / fast with default bandwidth caps
similar to HDFS i.e:
SLOW => 10MB/s
Medium => 100MB/s
Fast => 10GB/s
And will also have the following defaults for each profile:
Max datanode involvement cap
SLOW => 10%
Medium => 20%
Fast => 40%
Default max entering target
SLOW => 10 GB
Medium => 26 GB
Fast => 100 GB
Given the cluster summary from analyzer(HDDS-16173) and either default values
or values given by user for configs -> timeouts, balancing interval, threshold,
include/exclude nodes,
compute per profile: bytes to move, bytes to move per iteration, estimated
iterations, cycle time, and total estimated duration.
Formulas are size-based approximations — they do not simulate which containers
would move.
{code:java}
perNodeData = balancingBandwidth × moveReplicationTimeout_seconds
upperBound = min(perNodeData x sourceCount,perNodeData x
targetCount,MAX_SIZE_CEILING, BytesToMove)
maxSizeToMovePerIteration = max(containerSize + 1, upperBound)
estimatedIterations = ceil(bytesToMove / maxSizeToMovePerIteration)
iterationsWithBuffer = ceil(estimatedIterations × 1.3) // 1.3 for buffer
cycleTime = planning + moveTimeout + balancingInterval
totalTime = estimatedIterations × cycleTime
{code}
Validation: there must be at least one source and one target, bytes to move
must be positive and larger than container size, and timeout values must be
internally consistent. A balanced cluster should fail with a clear message here.
was:
The assessment command in HDDS-16174 tells users how much cluster imbalance
there is. The next step is estimating how long balancing will take and how many
iterations it will need — without actually moving containers or calling SCM
planning logic.
Introduce three profiles slow / medium / fast with default bandwidth caps
similar to HDFS i.e:
SLOW => 10MB/s
Medium => 100MB/s
Fast => 10GB/s
And will also have the following defaults for each profile:
Max datanode involvement cap
SLOW => 10%
Medium => 20%
Fast => 40%
Default max entering target
SLOW => 10 GB
Medium => 26 GB
Fast => 100 GB
Given the cluster summary from analyzer(HDDS-16173) and either default values
or values given by user for configs -> timeouts, balancing interval, threshold,
include/exclude nodes,
compute per profile: bytes to move, bytes to move per iteration, estimated
iterations, cycle time, and total estimated duration.
Formulas are size-based approximations — they do not simulate which containers
would move.
{code:java}
perNodeData = balancingBandwidth × moveReplicationTimeout_seconds
upperBound = min(perNodeData x sourceCount,perNodeData x
targetCount,MAX_SIZE_CEILING, BytesToMove)
maxSizeToMovePerIteration = max(containerSize + 1, upperBound)
estimatedIterations = ceil(bytesToMove / maxSizeToMovePerIteration)
cycleTime = moveTimeout + balancingInterval
totalTime = estimatedIterations × cycleTime
{code}
Validation: there must be at least one source and one target, bytes to move
must be positive and larger than container size, and timeout values must be
internally consistent. A balanced cluster should fail with a clear message here.
> Add logic to estimate balancer duration and iteration count
> -----------------------------------------------------------
>
> Key: HDDS-16179
> URL: https://issues.apache.org/jira/browse/HDDS-16179
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Sreeja
> Assignee: Sreeja
> Priority: Major
>
> The assessment command in HDDS-16174 tells users how much cluster imbalance
> there is. The next step is estimating how long balancing will take and how
> many iterations it will need — without actually moving containers or calling
> SCM planning logic.
>
> Introduce three profiles slow / medium / fast with default bandwidth caps
> similar to HDFS i.e:
> SLOW => 10MB/s
> Medium => 100MB/s
> Fast => 10GB/s
> And will also have the following defaults for each profile:
> Max datanode involvement cap
> SLOW => 10%
> Medium => 20%
> Fast => 40%
> Default max entering target
> SLOW => 10 GB
> Medium => 26 GB
> Fast => 100 GB
> Given the cluster summary from analyzer(HDDS-16173) and either default values
> or values given by user for configs -> timeouts, balancing interval,
> threshold, include/exclude nodes,
> compute per profile: bytes to move, bytes to move per iteration, estimated
> iterations, cycle time, and total estimated duration.
> Formulas are size-based approximations — they do not simulate which
> containers would move.
> {code:java}
> perNodeData = balancingBandwidth × moveReplicationTimeout_seconds
> upperBound = min(perNodeData x sourceCount,perNodeData x
> targetCount,MAX_SIZE_CEILING, BytesToMove)
> maxSizeToMovePerIteration = max(containerSize + 1, upperBound)
> estimatedIterations = ceil(bytesToMove / maxSizeToMovePerIteration)
> iterationsWithBuffer = ceil(estimatedIterations × 1.3) // 1.3 for buffer
> cycleTime = planning + moveTimeout + balancingInterval
> totalTime = estimatedIterations × cycleTime
> {code}
> Validation: there must be at least one source and one target, bytes to move
> must be positive and larger than container size, and timeout values must be
> internally consistent. A balanced cluster should fail with a clear message
> here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
