sidshas03 commented on code in PR #14697:
URL: https://github.com/apache/cloudstack/pull/14697#discussion_r4607927421


##########
agent/src/main/java/com/cloud/agent/Agent.java:
##########
@@ -663,6 +663,35 @@ protected void reconnect(final Link link, String 
preferredMSHost, boolean forTra
         logger.info("Connected to the host: {}", shell.getConnectedHost());
     }
 
+    protected void waitForCommandsAndReconnect(final Link link) {
+        int inProgress = commandsInProgress.get();
+        if (inProgress > 0) {
+            logger.warn("Agent disconnected with {} command(s) still in 
progress. Waiting for them to complete before reconnecting.", inProgress);
+
+            long maxWaitMs = 300000;
+            long waitedMs = 0;
+            while (commandsInProgress.get() > 0 && waitedMs < maxWaitMs) {
+                try {
+                    Thread.sleep(1000);
+                } catch (InterruptedException e) {
+                    Thread.currentThread().interrupt();
+                    break;
+                }
+                waitedMs += 1000;
+            }
+
+            if (commandsInProgress.get() > 0) {
+                logger.warn("Timed out waiting for {} command(s) to complete 
after {} ms. Proceeding with reconnect — "
+                        + "in-progress commands (e.g. snapshot merges) may not 
have completed.", commandsInProgress.get(), waitedMs);
+            } else {
+                logger.info("All in-progress commands completed. Proceeding 
with reconnect.");
+            }
+        }
+
+        logger.info("Reconnecting instead of exiting to avoid losing in-flight 
work (e.g. snapshot overlays).");
+        reconnect(link);

Review Comment:
   Fair point. MS drops the connection context the moment it sees agent 
disconnect, so waiting for responses nobody is listening for does not help. 
Simplified this stripped the polling loop, just reconnect directly. Still 
better than the old System.exit(1) since it does not kill in-flight libvirt 
operations. But the real safety net is the marker-based reconciliation on 
startup.



##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java:
##########
@@ -2528,10 +2565,29 @@ private SnapshotObjectTO 
takeFullVolumeSnapshotOfRunningVm(CreateObjectCommand c
 
             String convertResult = 
convertBaseFileToSnapshotFileInStorageDir(ObjectUtils.defaultIfNull(secondaryPool,
 primaryPool), disk, snapshotPath, directoryPath, volume, cmd.getWait());
 
-            resource.mergeDeltaIntoBaseFile(vm, diskLabel, diskPath, null, 
true, snapshotName, volume, conn);
+            ReentrantLock vmLock = getVmBlockJobLock(vmName);
+            boolean lockAcquired = false;
+            try {
+                lockAcquired = vmLock.tryLock(incrementalSnapshotTimeout, 
TimeUnit.MILLISECONDS);
+                if (!lockAcquired) {
+                    throw new CloudRuntimeException(String.format("Could not 
acquire block job lock for VM [%s] within timeout [%d ms]. Another block job 
(snapshot/backup) may be in progress.", vmName, incrementalSnapshotTimeout));
+                }
+
+                resource.mergeDeltaIntoBaseFile(vm, diskLabel, diskPath, null, 
true, snapshotName, volume, conn);
+            } catch (InterruptedException e) {
+                Thread.currentThread().interrupt();
+                throw new CloudRuntimeException("Interrupted while waiting for 
block job lock for VM [" + vmName + "]", e);
+            } finally {
+                if (lockAcquired) {
+                    vmLock.unlock();
+                }
+            }
+
+            removeSnapshotMarker(primaryPool, diskPath, snapshotName);
 
             validateConvertResult(convertResult, snapshotPath);
         } catch (LibvirtException e) {
+            removeSnapshotMarker(primaryPool, diskPath, snapshotName);

Review Comment:
   Correct, removed that call. If a LibvirtException happens the overlay may 
still be on disk, and removing the marker means reconciliation will never find 
it on restart.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to