ArafatKhan2198 commented on PR #6742:
URL: https://github.com/apache/ozone/pull/6742#issuecomment-2154152459
Thanks for the comment, @raju-balpande . Recon does not work the way you
have described. It is not responsible for creating pipeline IDs. SCM (Storage
Container Manager) is actually responsible for creating pipelines and their
related information along with their UUIDs, which have been thoroughly tested
and are robust. SCM will not create a duplicate pipeline ID. If it did, it
would affect a lot of other things, not just Recon.
Pipelines are created by SCM and then registered in Recon. We have a
`PipelineSyncTask` in Recon that runs every 5 minutes. This task makes an RPC
call to SCM every 5 minutes to fetch the latest set of pipelines and adds or
removes pipelines as needed. Additionally, the ReconContainerManager can add a
new pipeline. If Recon receives a container report for a new container whose
pipeline does not exist, it registers the new pipeline by making an RPC call to
SCM, fetches the pipeline information, and then adds it.
The addPipeline method in ReconPipelineManager can be called from two
entities. The reason for the duplicate pipeline error was that in the
ReconContainerManager code, we first check if the pipeline exists. If it does
not, we proceed to add it. However, as soon as the existence check passed, the
PipelineSyncTask had already added the pipeline, causing the duplicate pipeline
error.
```
// Check if the pipeline is present in Recon
if (!pipelineManager.containsPipeline(pipelineID)) {
// Pipeline is not present, add it first.
LOG.info("Adding new pipeline {} from SCM.", pipelineID);
reconPipelineManager.addPipeline(containerWithPipeline.getPipeline());
}
```
To address this, we added a final check inside the addPipeline method of
ReconPipelineManager. Here, we acquire a write lock, check if the pipeline
exists, and then add the pipeline if it does not. This ensures that the
exception is not thrown and the appropriate log message is delivered.
```
@VisibleForTesting
public void addPipeline(Pipeline pipeline)
throws IOException {
acquireWriteLock();
try {
if (!containsPipeline(pipeline.getId())) {
getStateManager().addPipeline(
pipeline.getProtobufMessage(ClientVersion.CURRENT_VERSION));
}
} finally {
releaseWriteLock();
}
}
```
Recon operates on eventual consistency, meaning it will eventually update
and correct itself when state changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
