umamaheswararao commented on PR #3361:
URL: https://github.com/apache/ozone/pull/3361#issuecomment-1119844168
### Implementation Level Design Choices for Recovered Containers Storage
Discussion with @guihecheng , @sodonnel , @kaijchen and @umamaheswararao
In today's offline ( online zoom :-) ) discussion, we have discussed the
following:
With the current discussions we have few options to store the recovered
containers at DN.
### 1. Creating the recovered containers locally in a tempStore Service:
**Advantages:**
1. This option gives some advantage as we don’t need to stream the data to
remote nodes. All containers first recovered locally and transferred. So, less
failure probability.
2. When transferring if target nodes failed, then with additional buffer
targets, we can transfer the same replica to other target.
**Disadvantages:**
1. It would be slow.
3. More disk IO as we will need to write locally and read back.
4. Replica downloaders need to understand the tempstore container
structure when downloading.
### 2. Creating containers remotely and use the different state “Recovering”
**Advantages:**
1. We can reuse more code in the write path at DN.
**Disadvantages:**
1. Failure handling is tricky. Need additional daemon services/other
mechanisms to clean the previously failed and still in “Recovering” state
containers.
2. Possibility of more failures due to NW transfer along the recovery.
We may need to discard the work done on failures.
Refer below section [Further details on the Recovering State on DNs.]
for additional information.
### 3. Use “TempStore” service, but still transfer the replicas remotely.
**Advantages:**
1. We can have our own metadata structure, ex: in memory.
**Disadvantages:**
1. Some new interfaces and new implementations, so we need more test
coverage.
2. Same as #2
3. May not reuse the existing write paths.
### 4. Create containers locally, but use “Recovering” State and create
containers with the name replicaIndex.
**Advantages:**
1. We may be able to reuse the code and can come to a stable state quickly.
2. Less probability for intermittent NW failures.
3. With fewer modifications ReplicaDownloader may work.
**Disadvantages:**
1. We need to modify it to allow containers to have different names.
2. We still need some special scrubber services to clean up the failed
container. Relatively easy as Coordinator only creates those containers, so if
it restarts, either it can catch up/ we can discard them.
3. Same as 1.1: It would be slow.
5. Same as 1.2: More disk IO as we will need to write locally and read back.
**Additional Discussions:**
Other arguments we also have that, in a good NW configuration machine, we
don’t need to worry about NW failure probability. But on average today’s
cluster deployments we still see NW issues though.
We also should be able to transfer the replicas as is - similar to Ratis
replication as with decommission we may need this. But I think if we schedule
the ReplicateContainer command instead of ReconstructECCommand, that should
work.
### Further details on the Recovering State on DNs.
Potentially this could work as follows. The coordinator issues a
“createContainer” call with a flag to indicate that it is a recovering
container.
The DN stores this container in the usual place, but keeps track of it in a
ECRecoveryMonitor. We skip sending an ICR or including it in any container
reporting.
If the DN is restarted when a container is recovering, we should just remove
any recovering containers, as the coordinator will have failed anyway.
The coordinator then writes to the container as usual and the DN stores the
chunks as usual.
When all chunks are recovered, the coordinator issues a new
“completeRecovery” call to the datanode. This will trigger an ICR etc.
If the coordinator fails for some reason, we need to clean up. The
ECRecoveryMonitor on the datanode can scan the set of recovering containers
looking for progress. Eg if the last write was longer than some time threshold,
assume the coordinator has failed and remove the container. If the coordinator
comes back again and tries to write, it will get an container does not exist
error.
Another edge case is that the recovery coordinator fails and is rescheduled,
picking the same host as the target. It tries to create the recovering
container and finds it already exists. In that case, we can just remove the
recovering container and create a new empty one instead.
A further enhancement may allow us to read what is in the partly recovered
container and restart recovered, but that is probably too complex for day 1.
If we build as described above, there are no changes needed on SCM. We just
need to handle a few areas where ICRs are sent, and create the new
ECRecoveryMonitor in the Datanodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
