errose28 commented on PR #3360:
URL: https://github.com/apache/ozone/pull/3360#issuecomment-1297759990
Ok we can continue with the proposal to remove the deleted containers from
RocksDB. Does this description line up with what is being proposed?
- When replication manager finds an empty container to delete, it will send
the delete commands to replicas and remove the container from in memory and the
DB.
- If datanodes miss the command, the container will be deleted if empty
on the next FCR by the container report handler.
- When SCM gets a report for a container that it does not have in the DB it
will delete it if it is empty (block count is 0), otherwise it will handle the
container based on the value of `hdds.scm.unknown-container.action`.
- DELETED containers already in SCM DB will remain there forever.
- If we want to get rid of them on startup, we would need a layout
feature for this as it would affect downgrades otherwise.
- This flow will not be behind a config key.
The catch here is that the a container will get removed from SCM if all
replicas in the current replica set are empty, but SCM cannot know about all
replicas that may have a copy of the container. As @JacksonYao287 pointed out
in [this
comment](https://github.com/apache/ozone/pull/3360#issuecomment-1119204766), an
old replica whose container has been deleted could resurface, and have blocks
that already got deleted and cleared. If we are only deleting unknown empty
containers, then this container would not get deleted. However, deleting
unknown non-empty containers seems dangerous. Removal of these containers might
have to wait until we have orphan block cleanup on Recon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
