ashishkumar50 commented on code in PR #10000:
URL: https://github.com/apache/ozone/pull/10000#discussion_r3035999302
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerManagerImpl.java:
##########
@@ -242,12 +262,75 @@ private ContainerInfo createContainer(Pipeline pipeline,
String owner)
return containerInfo;
}
+ /**
+ * Check if a pipeline has sufficient space after considering pending
allocations.
+ * Tracks containers scheduled but not yet written to DataNodes, preventing
over-allocation.
+ *
+ * @param pipeline The pipeline to check
+ * @return true if sufficient space is available, false otherwise
+ */
+ private boolean hasSpaceAfterPendingAllocations(Pipeline pipeline) {
+ try {
+ for (DatanodeDetails node : pipeline.getNodes()) {
+ // Get DN's storage statistics
+ DatanodeInfo datanodeInfo = pipelineManager.getDatanodeInfo(node);
+ if (datanodeInfo == null) {
+ LOG.warn("DatanodeInfo not found for node {}", node.getUuidString());
+ return false;
+ }
+
+ List<StorageReportProto> storageReports =
datanodeInfo.getStorageReports();
+ if (storageReports == null || storageReports.isEmpty()) {
+ LOG.warn("No storage reports for node {}", node.getUuidString());
+ return false;
+ }
+
+ // Calculate total capacity and effective allocatable space
+ // For each disk, calculate how many containers can actually fit,
+ // since containers are written to individual disks, not spread across
them.
+ // Example: disk1=9GB, disk2=14GB with 5GB containers
+ // (1*5GB) + (2*5GB) = 15GB → actually 3 containers
+ long totalCapacity = 0L;
+ long effectiveAllocatableSpace = 0L;
+ for (StorageReportProto report : storageReports) {
+ totalCapacity += report.getCapacity();
+ long usableSpace = VolumeUsage.getUsableSpace(report);
+ // Calculate how many containers can fit on this disk
+ long containersOnThisDisk = usableSpace / maxContainerSize;
+ // Add effective space (containers that fit * container size)
+ effectiveAllocatableSpace += containersOnThisDisk * maxContainerSize;
+ }
+
+ // Get pending allocations from tracker
+ long pendingAllocations =
pendingContainerTracker.getPendingAllocationSize(node);
+
+ // Calculate effective remaining space after pending allocations
+ long effectiveRemaining = effectiveAllocatableSpace -
pendingAllocations;
+
+ // Check if there's enough space for a new container
+ if (effectiveRemaining < maxContainerSize) {
Review Comment:
No need of extra buffer here, as we are anyway going to give buffer in DN by
considering soft and hard limit. So in case of some overflow DN will accept it
until hard limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
