[
https://issues.apache.org/jira/browse/HDDS-17560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18134172#comment-18134172
]
Stephen O'Donnell commented on HDDS-17560:
------------------------------------------
Thanks for reporting this [~ssulav].
It seems like the problem is:
1. ReconNodeManager.register() catches InvalidTopologyException and returns
errorNodeNotPermitted to the DN. This was in place before HDDS-15482.
2. HDDS-15482 added errorNodeNotPermitted → DN SHUTDOWN logic in
RegisterEndpointTask, but didn't check whether the responding endpoint is
passive (Recon). RunningDatanodeState then sees any endpoint returning SHUTDOWN
and shuts down the whole datanode.
Before HDDS-15482, the errorNodeNotPermitted from Recon would fall through
throw a RuntimeException, and get caught as a task execution error — logged but
harmless. After HDDS-15482, it hits the new shutdown path.
Should be a fairly easy fix by telling it to treat errors from recon as "do not
shutdown".
> DN going down on Recon failing to register, error reported as ZDU version
> mismatch
> ----------------------------------------------------------------------------------
>
> Key: HDDS-17560
> URL: https://issues.apache.org/jira/browse/HDDS-17560
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.1
> Reporter: Soumitra Sulav
> Assignee: Stephen O'Donnell
> Priority: Critical
>
> DN also register to Recon, and trigger shutdown on failure. This shutdown is
> added as per ZDU feature in DN,
> HDDS-15482. Add fencing based on datanode versions to SCM and Recon.
> From this, if any bad follower or Recon, it can cause DN to be down.
> Please check and share the solution how it should behave.
>
> Log triggering shutdown, and this response is from Recon:
>
> {code:java}
> 2026-10-21 17:27:28,942 ERROR
> [8588d42a-db42-469c-a0c8-3475f8e3cea3-EndpointStateMachineTaskThread-ccycloud-5.quasar-poumbe.root.comops.site:9891/domain.hadoop.site/10.00.000.00-0
>
> ]-org.apache.hadoop.ozone.container.common.states.endpoint.RegisterEndpointTask:
> SCM rejected this datanode's registration with software version ZDU (100)
> and apparent version ZDU (100). This datanode will shut down. Check SCM logs
> for details and verify that the datanode software version is compatible with
> the cluster. {code}
>
> Recon log:
> {code:java}
> 2026-10-21 17:27:15,820 ERROR
> [main]-org.apache.hadoop.ozone.recon.scm.ReconNodeManager:
> InvalidTopologyException error occurred : Failed to add
> /default/c0fde83e-add0-4e5c-9974-e4efd00544f7: Its path depth is not 4
> 2026-10-21 17:27:15,820 INFO
> [main]-org.apache.hadoop.ozone.recon.ReconContext: Update healthStatus of
> Recon from false to false.
> {code}
>
> There are basically two issues :
> 1. Reported incorrect log on any sort of exception and seems like an upgrade
> failure.
> 2. Failure/Exception on DN registry on any non-leader SCM component can lead
> to DN shutdown exception.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
