Attila Doroszlai created HDDS-5312:
--------------------------------------
Summary: Intermittent failure in SCM Ratis integration test
Key: HDDS-5312
URL: https://issues.apache.org/jira/browse/HDDS-5312
Project: Apache Ozone
Issue Type: Bug
Components: SCM HA, test
Affects Versions: 1.2.0
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai
Some integration tests intermittently fail due to mini cluster not existing
safe mode within 2 minutes timeout. The problem is that pipeline creation
interval is also 2 minutes. It may happen that pipeline is created only while
the cluster is being shut down due to timeout.
{noformat:title=https://github.com/elek/ozone-build-results/blob/master/2021/06/02/8191/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot-output.txt}
2021-06-02 03:21:03,005 [RatisPipelineUtilsThread - 0] WARN
pipeline.PipelinePlacementPolicy
(PipelinePlacementPolicy.java:filterViableNodes(151)) - Pipeline creation
failed due to no sufficient healthy datanodes. Required 3. Found 2.
...
2021-06-02 03:21:04,007 [Listener at 127.0.0.1/40677] INFO
ozone.MiniOzoneClusterImpl
(MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(224)) - Nodes are
ready. Got 3 of 3 DN Heartbeats.
...
2021-06-02 03:22:59,107 [Listener at 127.0.0.1/40677] INFO
ozone.MiniOzoneClusterImpl (MiniOzoneClusterImpl.java:shutdown(443)) - Shutting
down the Mini Ozone Cluster
...
2021-06-02 03:23:03,031
[6d4e3dd1-e161-4c07-861b-817db46a0427@group-0D81E0660BF9-StateMachineUpdater]
INFO pipeline.PipelineStateManager
(PipelineStateManagerV2Impl.java:addPipeline(101)) - Created pipeline Pipeline
... RATIS/THREE ...
{noformat}
{noformat:title=https://github.com/elek/ozone-build-results/blob/master/2021/06/02/8191/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot.txt}
org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot Time elapsed: 146.994 s <<<
ERROR!
java.util.concurrent.TimeoutException:
...
at
org.apache.hadoop.ozone.MiniOzoneClusterImpl.waitForClusterToBeReady(MiniOzoneClusterImpl.java:217)
{noformat}
Related test failures:
{noformat}
2021/05/26/8113/it-client/hadoop-ozone/integration-test/org.apache.hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis.txt
2021/05/26/8118/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.scm.TestStorageContainerManagerHA.txt
2021/05/27/8142/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot.txt
2021/05/30/8164/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.om.TestOzoneManagerRestInterface.txt
2021/05/31/8166/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMSnapshot.txt
2021/05/31/8177/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.pipeline.TestPipelineClose.txt
2021/06/02/8191/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot.txt
2021/06/02/8193/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.scm.TestStorageContainerManagerHA.txt
2021/06/02/8211/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMSnapshot.txt
2021/06/02/8217/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot.txt
2021/06/07/8299/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot.txt
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
