Bharat Viswanadham created HDDS-3067:
----------------------------------------
Summary: Fix Bug in Scrub Pipeline
Key: HDDS-3067
URL: https://issues.apache.org/jira/browse/HDDS-3067
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham
Currently, the scrubber is run as part of create pipeline.
When SCM is started, scrubber is coming up and cleaning up all the containers
in SCM. Because when loading pipelines, the pipelineCreationTimeStamp is set
from when the pipeline is created.
Because of this, below condition is satisfied and destroying all the pipelines
when SCM is restarted. This can be easily reproduced start SCM, wait for 10
minutes and restart SCM.
{code:java}
List<Pipeline> needToSrubPipelines = stateManager.getPipelines(type, factor,
Pipeline.PipelineState.ALLOCATED).stream()
.filter(p -> currentTime.toEpochMilli() - p.getCreationTimestamp()
.toEpochMilli() >= pipelineScrubTimeoutInMills)
.collect(Collectors.toList());
for (Pipeline p : needToSrubPipelines) {
LOG.info("srubbing pipeline: id: " + p.getId().toString() +
" since it stays at ALLOCATED stage for " +
Duration.between(currentTime, p.getCreationTimestamp()).toMinutes() +
" mins.");
finalizeAndDestroyPipeline(p, false);
}{code}
*Log showing scrubbing of pipeline*
{code:java}
2020-02-20 12:42:18,946 [RatisPipelineUtilsThread] INFO
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: srubbing pipeline: id:
PipelineID=35dff62d-9bfa-449b-b6e8-6f00cc8c1b6e since it stays at ALLOCATED
stage for -1003 mins.{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
