[
https://issues.apache.org/jira/browse/HDDS-16387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navin Kumar updated HDDS-16387:
-------------------------------
Description:
ContainerBalancer#start() and ContainerBalancer#startBalancer() assign
{{startedAt = OffsetDateTime.now()}} at the very top of the method, *before*
state/configuration validation runs. If a start attempt is rejected during
validation, startedAt is still advanced even though no new balancing run
actually begins.
After a previous run has completed (so stoppedAt is set), a subsequent rejected
start pushes startedAt *past* stoppedAt. This yields a negative "balancing
duration".
{code}
bash-5.1$ ozone admin containerbalancer start -d 10
Failed to start Container Balancer.
max-datanodes-percentage-to-involve-per-iteration=10 allows at most 0
datanode(s) per iteration with 5 eligible datanode(s), but at least 2 are
required for a source and target datanode pair.
{code}
{code}
bash-5.1$ ozone admin containerbalancer status --verbose
ContainerBalancer is Not Running.
Stop reason: INITIALIZATION_FAILED
Message: Failed to initialize a container balancer iteration. Details:
iteration number 1, Did not find any unbalanced Datanodes.
Started at: 2026-09-05 19:12:37
Stopped at: 2026-09-05 15:25:14
java.lang.IllegalStateException: Provided duration is incorrect: PT-3H-47M-23S
at
org.apache.hadoop.hdds.util.DurationUtil.getPrettyDuration(DurationUtil.java:49)
at
org.apache.hadoop.hdds.scm.cli.ContainerBalancerStatusSubcommand.printVerboseStatusInfo(ContainerBalancerStatusSubcommand.java:98)
at
org.apache.hadoop.hdds.scm.cli.ContainerBalancerStatusSubcommand.execute(ContainerBalancerStatusSubcommand.java:75)
at
org.apache.hadoop.hdds.scm.cli.ScmSubcommand.call(ScmSubcommand.java:39)
at
org.apache.hadoop.hdds.scm.cli.ScmSubcommand.call(ScmSubcommand.java:29)
at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at
picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
at
picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
at org.apache.hadoop.ozone.shell.Shell.lambda$execute$0(Shell.java:104)
at
org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:447)
at
org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:436)
at
org.apache.hadoop.hdds.tracing.TracingUtil.execute(TracingUtil.java:138)
at org.apache.hadoop.ozone.shell.Shell.execute(Shell.java:103)
at picocli.CommandLine.execute(CommandLine.java:2170)
at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:92)
at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:84)
at org.apache.hadoop.ozone.admin.OzoneAdmin.main(OzoneAdmin.java:36)
{code}
was:
ContainerBalancer#start() and ContainerBalancer#startBalancer() assign
{\{startedAt = OffsetDateTime.now()}} at the very top of the method, *before*
state/configuration validation runs. If a start attempt is rejected during
validation, startedAt is still advanced even though no new balancing
run actually begins.
After a previous run has completed (so stoppedAt is set), a subsequent
rejected start pushes startedAt *past* stoppedAt. This yields a negative
"balancing duration".
ContainerBalancer#start() and ContainerBalancer#startBalancer() assign
{\{startedAt = OffsetDateTime.now()}} at the very top of the method, *before*
state/configuration validation runs. If a start attempt is rejected during
validation (invalid config, SCM in safe mode, non-leader SCM, or balancer
already running), startedAt is still advanced even though no new balancing
run actually begins.
After a previous run has completed (so stoppedAt is set), a subsequent
rejected start pushes startedAt *past* stoppedAt. This yields a negative
"balancing duration".
> ContainerBalancer status --verbose throws IllegalStateException after a
> rejected start
> --------------------------------------------------------------------------------------
>
> Key: HDDS-16387
> URL: https://issues.apache.org/jira/browse/HDDS-16387
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Reporter: Navin Kumar
> Assignee: Navin Kumar
> Priority: Major
>
> ContainerBalancer#start() and ContainerBalancer#startBalancer() assign
> {{startedAt = OffsetDateTime.now()}} at the very top of the method, *before*
> state/configuration validation runs. If a start attempt is rejected during
> validation, startedAt is still advanced even though no new balancing run
> actually begins.
>
> After a previous run has completed (so stoppedAt is set), a subsequent
> rejected start pushes startedAt *past* stoppedAt. This yields a negative
> "balancing duration".
> {code}
> bash-5.1$ ozone admin containerbalancer start -d 10
>
> Failed to start Container Balancer.
> max-datanodes-percentage-to-involve-per-iteration=10 allows at most 0
> datanode(s) per iteration with 5 eligible datanode(s), but at least 2 are
> required for a source and target datanode pair.
> {code}
> {code}
> bash-5.1$ ozone admin containerbalancer status --verbose
> ContainerBalancer is Not Running.
> Stop reason: INITIALIZATION_FAILED
> Message: Failed to initialize a container balancer iteration. Details:
> iteration number 1, Did not find any unbalanced Datanodes.
> Started at: 2026-09-05 19:12:37
> Stopped at: 2026-09-05 15:25:14
> java.lang.IllegalStateException: Provided duration is incorrect: PT-3H-47M-23S
> at
> org.apache.hadoop.hdds.util.DurationUtil.getPrettyDuration(DurationUtil.java:49)
> at
> org.apache.hadoop.hdds.scm.cli.ContainerBalancerStatusSubcommand.printVerboseStatusInfo(ContainerBalancerStatusSubcommand.java:98)
> at
> org.apache.hadoop.hdds.scm.cli.ContainerBalancerStatusSubcommand.execute(ContainerBalancerStatusSubcommand.java:75)
> at
> org.apache.hadoop.hdds.scm.cli.ScmSubcommand.call(ScmSubcommand.java:39)
> at
> org.apache.hadoop.hdds.scm.cli.ScmSubcommand.call(ScmSubcommand.java:29)
> at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
> at picocli.CommandLine.access$1500(CommandLine.java:148)
> at
> picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
> at
> picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
> at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
> at
> org.apache.hadoop.ozone.shell.Shell.lambda$execute$0(Shell.java:104)
> at
> org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:447)
> at
> org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:436)
> at
> org.apache.hadoop.hdds.tracing.TracingUtil.execute(TracingUtil.java:138)
> at org.apache.hadoop.ozone.shell.Shell.execute(Shell.java:103)
> at picocli.CommandLine.execute(CommandLine.java:2170)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:92)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:84)
> at org.apache.hadoop.ozone.admin.OzoneAdmin.main(OzoneAdmin.java:36)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
