[
https://issues.apache.org/jira/browse/HDDS-15688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krishna Kumar Asawa reassigned HDDS-15688:
------------------------------------------
Assignee: Gargi Jaiswal
> DiskBalancer reverts to RUNNING when diskBalancer.info persist fails in
> nodeStateUpdated() during decommission/maintenance
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-15688
> URL: https://issues.apache.org/jira/browse/HDDS-15688
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode
> Affects Versions: 2.3.0
> Reporter: Arun Sarin
> Assignee: Gargi Jaiswal
> Priority: Major
> Attachments: repro-diskbalancer-decommission-persist-failure.sh,
> repro_diskbalancer_decommission_persist_20260626_131028.txt,
> repro_diskbalancer_decommission_persist_20260626_131028_datanode_filtered.log,
> repro_diskbalancer_decommission_persist_20260626_131028_datanode_full.log,
> repro_diskbalancer_decommission_persist_20260626_131028_scm_filtered.log,
> repro_diskbalancer_decommission_persist_20260626_131028_scm_full.log
>
>
> Hi [~gargijaiswal] / [~sammichen] ,
> Flagging a safety issue in the persist-failure path added around
> {{nodeStateUpdated()}} in {{DiskBalancerService}}
> When a DN enters {{DECOMMISSIONING}} or {{{}ENTERING_MAINTENANCE{}}}, the
> service correctly sets {{operationalState = PAUSED}} and calls
> {{{}writeDiskBalancerInfoTo(){}}}. If that throws {{{}IOException{}}}, the
> catch block reverts to {{originalServiceState}} — which is {{RUNNING}} when
> we were trying to pause for decommission. {{getTasks()}} then keeps
> scheduling moves on a node that is already leaving service.
> The revert-to-last-persisted-state logic makes sense for {{PAUSED → RUNNING}}
> (resume fails → stay paused). It is unsafe for {{RUNNING → PAUSED}} (pause
> fails → must stay paused in memory).
> [https://github.com/apache/ozone/blob/29b2ad5fcd40642c77f771128e7f0f7c4fe88e42/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java#L853-L864]
> In production , same can happen in any of the below cases:
> 1. Decommission during metadata volume pressure
> User decommissions a DN while the metadata disk is nearly full (RocksDB
> container DBs, SCM heartbeat state, {{{}diskBalancer.info{}}}, etc. on the
> same volume). The atomic write creates {{diskBalancer.info.tmp}} but the
> rename to {{diskBalancer.info}} fails with no space. DN is already
> {{{}DECOMMISSIONING{}}}; DiskBalancer reverts to {{RUNNING}} and may still be
> moving containers off that node while SCM is draining it.
> 2. Datanode in Maintenance window
> Node put into {{ENTERING_MAINTENANCE}} during a disk/hardware incident on the
> metadata mount. Same path as decommission - pause is attempted, persist
> fails, revert to {{{}RUNNING{}}}. Maintenance is meant to stop user I/O and
> background work; DiskBalancer keeps running.
> 3. Race with concurrent admin action
> {{ozone admin datanode diskbalancer start}} (or stop) and decommission
> overlap. Both paths persist via {{applyDiskBalancerInfo()}} /
> {{{}nodeStateUpdated(){}}}. A failed write on the decommission pause path
> still reverts to {{RUNNING}} regardless of what the admin command intended.
> h3. Repro (confirmed on master)
> Compose cluster, 5 DNs, RF=3, {{{}ozone-2.3.0-SNAPSHOT{}}}.
> # Start DiskBalancer on IN_SERVICE DN → RPC {{RUNNING}}
> # Break persist: {{rm -rf /data/metadata/diskBalancer.info && mkdir
> /data/metadata/diskBalancer.info}}
> # {{ozone admin datanode decommission <dn-ip>:19864}}
> # DN reaches {{{}DECOMMISSIONING{}}}; DiskBalancer RPC still {{RUNNING}}
> *Automated script[To repro]:*
> *[^repro-diskbalancer-decommission-persist-failure.sh]*
> h3. Expected vs actual
> || ||Expected||Actual||
> |DN op state after decommission|{{DECOMMISSIONING}}|{{DECOMMISSIONING}} ✓|
> |DiskBalancer in-memory / RPC|{{PAUSED}}|{{RUNNING}} ✗|
> h3. Suggested fix
> On persist failure in {{{}nodeStateUpdated(){}}}:
> * If we were pausing ({{{}operationalState == PAUSED{}}} after transition):
> keep PAUSED in memory; log retain despite failure
> * If we were resuming ({{{}operationalState == RUNNING{}}} after
> transition): revert to PAUSED (current revert behavior is already safe here)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
