sodonnel commented on PR #3360: URL: https://github.com/apache/ozone/pull/3360#issuecomment-1117404215
I think it is reasonable to remove containers that are empty. There isn't a good reason to keep them beyond debugging and if a DN comes back up with a container on it, we should probably handle it as an unknown container and remove it. I know the default Ozone behaviour is to ignore unknown containers and not remove them. I understand why (we don't want to accidentality delete data), but I feel that over time the number of empty containers in a cluster will grow. Also if the cluster is out of safemode, it seems overly pessimistic to keep unknown containers. It may seem unrealistic that we will end up with a log of empty containers, but if we have Hive workloads for example. They can have many phases and stage the temp data from between jobs on the cluster. This will fill containers and then they will be purged. A job could stage 100's of GB potentially in a large multi-table join. Related to this, and a problem I foresee, is that a cluster running these Hive type workloads may end up filling containers partly with temp data and partly with real data. The containers get closed. The tmp data gets removed and we end up with lots of small containers that are never totally empty, but may have much less than 5GB of data in them. Then the SCM memory will get used with not just empty container references, but far more containers than we expected (small container problem). We ultimately need to solve that small container problem by merging containers, but there is at least one major problem. We cannot easily update OM with the new containerID. One suggestion was to create a mapping in SCM. Another idea may be to find the keys associated with a container in Recon (reverse index of containerID -> key) and then update OM. For this PR, perhaps removing the deleted container reference in SCM could be tied to the same config that lets us delete or not delete unknown containers, so it either logs a warning or deletes the SCM reference depending on the value? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
