[
https://issues.apache.org/jira/browse/HDDS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-3067:
-------------------------------------
Fix Version/s: 0.6.0
0.5.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix Bug in Scrub Pipeline causing destory pipelines after SCM restart
> ---------------------------------------------------------------------
>
> Key: HDDS-3067
> URL: https://issues.apache.org/jira/browse/HDDS-3067
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
> Reporter: Bharat Viswanadham
> Assignee: Bharat Viswanadham
> Priority: Blocker
> Labels: OMHATest, pull-request-available
> Fix For: 0.5.0, 0.6.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, the scrubber is run as part of create pipeline.
> When SCM is started, scrubber is coming up and cleaning up all the containers
> in SCM. Because when loading pipelines, the pipelineCreationTimeStamp is set
> from when the pipeline is created.
>
> Because of this, below condition is satisfied and destroying all the
> pipelines when SCM is restarted. This can be easily reproduced start SCM,
> wait for 10 minutes and restart SCM.
>
> {code:java}
> List<Pipeline> needToSrubPipelines = stateManager.getPipelines(type, factor,
> Pipeline.PipelineState.ALLOCATED).stream()
> .filter(p -> currentTime.toEpochMilli() - p.getCreationTimestamp()
> .toEpochMilli() >= pipelineScrubTimeoutInMills)
> .collect(Collectors.toList());
> for (Pipeline p : needToSrubPipelines) {
> LOG.info("srubbing pipeline: id: " + p.getId().toString() +
> " since it stays at ALLOCATED stage for " +
> Duration.between(currentTime, p.getCreationTimestamp()).toMinutes() +
> " mins.");
> finalizeAndDestroyPipeline(p, false);
> }{code}
>
> *Log showing scrubbing of pipeline*
>
> {code:java}
> 2020-02-20 12:42:18,946 [RatisPipelineUtilsThread] INFO
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: srubbing pipeline:
> id: PipelineID=35dff62d-9bfa-449b-b6e8-6f00cc8c1b6e since it stays at
> ALLOCATED stage for -1003 mins.{code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
