Hanisha Koneru created HDDS-3724:
------------------------------------
Summary: BackgroundPipelineCreator should not throw Pipeline
creation failed exception due to insufficient nodes
Key: HDDS-3724
URL: https://issues.apache.org/jira/browse/HDDS-3724
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Hanisha Koneru
BackgroundPipelineCreator runs a scheduler to keep creating pipelines if
possible. But in a healthy cluster where all the datanodes are part of
pipelines and there are no more nodes available for creating new pipelines, the
BackgroundPipelineCreator should not log an error message. In this scenario, it
is not an error that a pipeline couldn't be created.
SCM logs were flooded with "Failed to create pipeline" error messages because
of this:
{code:java}
2020-06-04 11:03:24,229 [RatisPipelineUtilsThread] ERROR
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
due to no sufficient healthy datanodes. Required 3. Found 1.
2020-06-04 11:05:24,231 [RatisPipelineUtilsThread] ERROR
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1
2020-06-04 11:05:24,231 [RatisPipelineUtilsThread] WARN
org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline creation
failed due to no sufficient healthy datanodes. Required 3. Found 1.
2020-06-04 11:05:24,231 [RatisPipelineUtilsThread] ERROR
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
due to no sufficient healthy datanodes. Required 3. Found 1.
2020-06-04 11:05:40,705 [ReplicationMonitor] INFO
org.apache.hadoop.hdds.scm.container.ReplicationManager: Replication Monitor
Thread took 0 milliseconds for processing 7 containers.
2020-06-04 11:07:24,234 [RatisPipelineUtilsThread] ERROR
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1
2020-06-04 11:07:24,234 [RatisPipelineUtilsThread] WARN
org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline creation
failed due to no sufficient healthy datanodes. Required 3. Found 1.
2020-06-04 11:07:24,234 [RatisPipelineUtilsThread] ERROR
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
due to no sufficient healthy datanodes. Required 3. Found 1.
2020-06-04 11:09:24,242 [RatisPipelineUtilsThread] ERROR
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1
2020-06-04 11:09:24,242 [RatisPipelineUtilsThread] WARN
org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline creation
failed due to no sufficient healthy datanodes. Required 3. Found 1.
2020-06-04 11:09:24,242 [RatisPipelineUtilsThread] ERROR
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
due to no sufficient healthy datanodes. Required 3. Found 1.
2020-06-04 11:10:40,700 [ReplicationMonitor] INFO
org.apache.hadoop.hdds.scm.container.ReplicationManager: Replication Monitor
Thread took 0 milliseconds for processing 7 containers.
2020-06-04 11:11:24,234 [RatisPipelineUtilsThread] ERROR
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1
2020-06-04 11:11:24,235 [RatisPipelineUtilsThread] WARN
org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline creation
failed due to no sufficient healthy datanodes. Required 3. Found 1.
2020-06-04 11:11:24,235 [RatisPipelineUtilsThread] ERROR
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
due to no sufficient healthy datanodes. Required 3. Found 1.
2020-06-04 11:13:24,240 [RatisPipelineUtilsThread] ERROR
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1
2020-06-04 11:13:24,240 [RatisPipelineUtilsThread] WARN
org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline creation
failed due to no sufficient healthy datanodes. Required 3. Found 1.
2020-06-04 11:13:24,240 [RatisPipelineUtilsThread] ERROR
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
due to no sufficient healthy datanodes. Required 3. Found 1.
2020-06-04 11:15:24,242 [RatisPipelineUtilsThread] ERROR
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
