sodonnel commented on PR #3360: URL: https://github.com/apache/ozone/pull/3360#issuecomment-1290598823
I can kind of see this both ways. If we take it to one extreme - if we want to ensure stuff is kept around for debugging, why are we letting the empty replicas get deleted? On a real cluster blocks will get deleted all the time, and we will end up with empty containers. Some bug (which I think we already had) could erroneously delete replicas it should not, but its also not scalable to keep all empty replicas forever. They will grow slowly but surely to a large number over months and years and cause problems in the reporting and report processing. So if we have deleted the replicas, what use is a ContainerInfo reference in SCM? Even if we have it, we cannot do anything with it - there is no replica for it. All it tells us is that a container once existed with that ID, and its empty. Its bad that it will use up memory slowly over time. Replication Manager will also need to check it on each pass. We can keep it in the Rocks DB table unloaded on SCM startup, but (I think) we will still need to read out the record and decide on whether or not to load it into memory. Even will millions of such references the overhead should be small - it will only slowdown the SCM startup by a tiny amount, but its still a small overhead. I'm in favor of just removing them, as I see little value in keeping them. We should certainly log they have been removed - of course the logs will roll off, but at least that is some evidence they have been removed for a reason. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
