errose28 commented on PR #3465:
URL: https://github.com/apache/ozone/pull/3465#issuecomment-1149305172
Although the tests were passing pretty consistently with the last set of
changes and a bit longer test timeout, I have made some minor changes that I
think will improve the test execution and finalization in a live cluster
overall. The case we are working with is when finalization is run soon after
SCM exits safemode and the datanodes have not yet finished leader elections for
their pipelines. In this case, the close commands sent by SCM may result in all
3 container replicas entering the CLOSING state but no datanode actually
submitting the close command through Ratis. The datanodes will not finalize
when containers are in the CLOSING state.
In the original SCM finalization implementation, the
`HealthyReadonlyNodeHandler` implementation (which is invoked when datanode
heartbeats to SCM with a lower MLV) contained this block:
```java
Set<PipelineID> pipelineIDs = nodeManager.getPipelines(datanodeDetails);
for (PipelineID id: pipelineIDs) {
LOG.info("Closing pipeline {} which uses HEALTHY READONLY datanode {}
",
id, datanodeDetails);
try {
pipelineManager.closePipeline(pipelineManager.getPipeline(id),
false);
} catch (IOException ex) {
LOG.error("Failed to close pipeline {} which uses HEALTHY READONLY "
+
"datanode {}: ", id, datanodeDetails, ex);
}
}
```
Since pipelines were closed normally (not force closed) when finalization
started, this would force close the already closed pipelines, moving the
CLOSING containers to QUASI_CLOSED and allowing finalization to pass.
In earlier versions of this patch, I changed this block:
```java
Set<PipelineID> pipelineIDs = nodeManager.getPipelines(datanodeDetails);
for (PipelineID id: pipelineIDs) {
try {
Pipeline pipeline = pipelineManager.getPipeline(id);
if (pipeline.getPipelineState() != Pipeline.PipelineState.CLOSED) {
LOG.warn("Closing pipeline {} which uses HEALTHY READONLY datanode
" +
"{}.", id, datanodeDetails.getUuidString());
pipelineManager.closePipeline(pipeline, true);
}
} catch (IOException ex) {
LOG.error("Failed to close pipeline {} which uses HEALTHY READONLY "
+
"datanode {}: ", id, datanodeDetails, ex);
}
}
```
Since the pipelines were already closed during finalization, the pipeline
close here became a no-op, and the CLOSING containers lingered on the
datanodes. I originally handled this in the test by making the
ReplicationManager run soon after SCM exited safemode to move the containers
from CLOSING to CLOSED, but this was still slow to finish, both in the test and
possibly in a real cluster as well.
In the most recent update, I have changed the block to this:
```java
Set<PipelineID> pipelineIDs = nodeManager.getPipelines(datanodeDetails);
for (PipelineID pipelineID : pipelineIDs) {
try {
Pipeline pipeline = pipelineManager.getPipeline(pipelineID);
LOG.info("Sending close command for pipeline {} in state {} which " +
"uses {} datanode {}. This will send close commands for its
" +
"containers.",
pipelineID, pipeline.getPipelineState(),
HddsProtos.NodeState.HEALTHY_READONLY,
datanodeDetails.getUuidString());
pipelineManager.closePipeline(pipeline, true);
} catch (IOException ex) {
LOG.error("Failed to close pipeline {} which uses HEALTHY READONLY "
+
"datanode {}: ", pipelineID, datanodeDetails, ex);
}
}
```
Now when a healthy readonly datanode heartbeats back to SCM, the close
pipeline command will queue close container commands for all containers in the
pipeline. For already CLOSED containers this will be a no-op, but for CLOSING
containers whose datanodes have since finished their pipeline leader election,
this will move them to CLOSED without needing to wait for the
ReplicationManager to start. This should allow finalization to proceed at the
same pace as before, but without the occasional quasi closed containers and
without test specific config changes for the replication manager.
I will again be doing repeated testing of the acceptance test on my fork,
and will update here when it appears stable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
