[
https://issues.apache.org/jira/browse/HDDS-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDDS-5363:
--------------------------------------
Fix Version/s: 1.2.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Datanode shutdown due to too many bad volumes in CI
> ---------------------------------------------------
>
> Key: HDDS-5363
> URL: https://issues.apache.org/jira/browse/HDDS-5363
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode
> Affects Versions: 1.2.0
> Reporter: Attila Doroszlai
> Assignee: Attila Doroszlai
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.2.0
>
>
> _Acceptance (secure)_ check is frequently failing, usually at S3 tests. The
> root cause is that datanodes are shut down due to too many "bad" volumes.
> {noformat:title=S3 Gateway log}
> INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0
> blocks. Requested 1 blocks
> {noformat}
> {noformat:title=SCM log}
> Pipeline creation failed due to no sufficient healthy datanodes. Required 3.
> Found 0.
> {noformat}
> {noformat:title=Datanode log}
> datanode_2 | 2021-06-19 13:26:08,010 [Periodic HDDS volume checker] INFO
> volume.StorageVolumeChecker: Scheduled health check for volume /data/hdds/hdds
> datanode_2 | 2021-06-19 13:36:08,013 [Periodic HDDS volume checker] WARN
> volume.StorageVolumeChecker: checkAllVolumes timed out after 600000 ms
> datanode_2 | 2021-06-19 13:36:08,014 [Periodic HDDS volume checker] WARN
> volume.MutableVolumeSet: checkAllVolumes got 1 failed volumes -
> [/data/hdds/hdds]
> datanode_2 | 2021-06-19 13:36:08,016 [Periodic HDDS volume checker] INFO
> volume.MutableVolumeSet: Moving Volume : /data/hdds/hdds to failed Volumes
> datanode_2 | 2021-06-19 13:36:08,016 [Periodic HDDS volume checker] ERROR
> statemachine.DatanodeStateMachine: DatanodeStateMachine Shutdown due to too
> many bad volumes, check hdds.datanode.failed.data.volumes.tolerated and
> hdds.datanode.failed.metadata.volumes.tolerated
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
