Attila Doroszlai created HDDS-5363:
--------------------------------------
Summary: Datanode shutdown due to too many bad volumes in CI
Key: HDDS-5363
URL: https://issues.apache.org/jira/browse/HDDS-5363
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Affects Versions: 1.2.0
Reporter: Attila Doroszlai
_Acceptance (secure)_ check is frequently failing, usually at S3 tests. The
root cause is that datanodes are shut down due to too many "bad" volumes.
{noformat:title=S3 Gateway log}
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0
blocks. Requested 1 blocks
{noformat}
{noformat:title=SCM log}
Pipeline creation failed due to no sufficient healthy datanodes. Required 3.
Found 0.
{noformat}
{noformat:title=Datanode log}
datanode_2 | 2021-06-19 13:26:08,010 [Periodic HDDS volume checker] INFO
volume.StorageVolumeChecker: Scheduled health check for volume /data/hdds/hdds
datanode_2 | 2021-06-19 13:36:08,013 [Periodic HDDS volume checker] WARN
volume.StorageVolumeChecker: checkAllVolumes timed out after 600000 ms
datanode_2 | 2021-06-19 13:36:08,014 [Periodic HDDS volume checker] WARN
volume.MutableVolumeSet: checkAllVolumes got 1 failed volumes -
[/data/hdds/hdds]
datanode_2 | 2021-06-19 13:36:08,016 [Periodic HDDS volume checker] INFO
volume.MutableVolumeSet: Moving Volume : /data/hdds/hdds to failed Volumes
datanode_2 | 2021-06-19 13:36:08,016 [Periodic HDDS volume checker] ERROR
statemachine.DatanodeStateMachine: DatanodeStateMachine Shutdown due to too
many bad volumes, check hdds.datanode.failed.data.volumes.tolerated and
hdds.datanode.failed.metadata.volumes.tolerated
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
