siddhantsangwan opened a new pull request, #8813: URL: https://github.com/apache/ozone/pull/8813
## What changes were proposed in this pull request? For a description of the problem, please see the jira. This pull request introduces a map in `ContainerReplicaPendingOps` that tracks `Datanode -> Size of containers that are pending replication to that Datanode, and the last time at which this entry was updated`. This information can be used by the Replication Manager to check how much data is inflight to a particular Datanode to determine whether it has disk space available for more containers. The map is updated when a replication completes or when it expires. Every time a replication is scheduled for a target Datanode (ie, the datanode that will receive the container replica), the map is updated by adding the size of the container to the Datanode's entry and also updating the `lastUpdatedTime` to the current time. When a replication completes, SCM gets to know this when handling the container report received from a target Datanode. The report handler notifies `ContainerReplicaPendingOps` via Container Manager. On completion, the size of that particular container needs to subtracted from the size of data scheduled for that Datanode in the map. If the result is zero, then the entry is removed from the map. There's also a thread that runs in intervals and checks whether any pending ops in `ContainerReplicaPendingOps` have expired. I've added handling here to subtract a container's size or completely remove an entry if it has expired. Expiration is determined using the `lastUpdatedTime` stored in the map and the `eventTimeout` configuration in Replication Manager. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-13437 ## How was this patch tested? Added unit tests. Single line changes had to be made to unrelated unit tests because the signatures of the constructor and some methods in `ContainerReplicaPendingOps` changed. This is a draft while CI in progress in my fork, but is ready for review. https://github.com/siddhantsangwan/ozone/actions/runs/16293535058 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
