sidshas03 commented on code in PR #14697:
URL: https://github.com/apache/cloudstack/pull/14697#discussion_r4607927421
##########
agent/src/main/java/com/cloud/agent/Agent.java:
##########
@@ -663,6 +663,35 @@ protected void reconnect(final Link link, String
preferredMSHost, boolean forTra
logger.info("Connected to the host: {}", shell.getConnectedHost());
}
+ protected void waitForCommandsAndReconnect(final Link link) {
+ int inProgress = commandsInProgress.get();
+ if (inProgress > 0) {
+ logger.warn("Agent disconnected with {} command(s) still in
progress. Waiting for them to complete before reconnecting.", inProgress);
+
+ long maxWaitMs = 300000;
+ long waitedMs = 0;
+ while (commandsInProgress.get() > 0 && waitedMs < maxWaitMs) {
+ try {
+ Thread.sleep(1000);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ break;
+ }
+ waitedMs += 1000;
+ }
+
+ if (commandsInProgress.get() > 0) {
+ logger.warn("Timed out waiting for {} command(s) to complete
after {} ms. Proceeding with reconnect — "
+ + "in-progress commands (e.g. snapshot merges) may not
have completed.", commandsInProgress.get(), waitedMs);
+ } else {
+ logger.info("All in-progress commands completed. Proceeding
with reconnect.");
+ }
+ }
+
+ logger.info("Reconnecting instead of exiting to avoid losing in-flight
work (e.g. snapshot overlays).");
+ reconnect(link);
Review Comment:
Fair point. MS drops the connection context the moment it sees agent
disconnect, so waiting for responses nobody is listening for does not help.
Simplified this stripped the polling loop, just reconnect directly. Still
better than the old System.exit(1) since it does not kill in-flight libvirt
operations. But the real safety net is the marker-based reconciliation on
startup.
##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java:
##########
@@ -2528,10 +2565,29 @@ private SnapshotObjectTO
takeFullVolumeSnapshotOfRunningVm(CreateObjectCommand c
String convertResult =
convertBaseFileToSnapshotFileInStorageDir(ObjectUtils.defaultIfNull(secondaryPool,
primaryPool), disk, snapshotPath, directoryPath, volume, cmd.getWait());
- resource.mergeDeltaIntoBaseFile(vm, diskLabel, diskPath, null,
true, snapshotName, volume, conn);
+ ReentrantLock vmLock = getVmBlockJobLock(vmName);
+ boolean lockAcquired = false;
+ try {
+ lockAcquired = vmLock.tryLock(incrementalSnapshotTimeout,
TimeUnit.MILLISECONDS);
+ if (!lockAcquired) {
+ throw new CloudRuntimeException(String.format("Could not
acquire block job lock for VM [%s] within timeout [%d ms]. Another block job
(snapshot/backup) may be in progress.", vmName, incrementalSnapshotTimeout));
+ }
+
+ resource.mergeDeltaIntoBaseFile(vm, diskLabel, diskPath, null,
true, snapshotName, volume, conn);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new CloudRuntimeException("Interrupted while waiting for
block job lock for VM [" + vmName + "]", e);
+ } finally {
+ if (lockAcquired) {
+ vmLock.unlock();
+ }
+ }
+
+ removeSnapshotMarker(primaryPool, diskPath, snapshotName);
validateConvertResult(convertResult, snapshotPath);
} catch (LibvirtException e) {
+ removeSnapshotMarker(primaryPool, diskPath, snapshotName);
Review Comment:
Correct, removed that call. If a LibvirtException happens the overlay may
still be on disk, and removing the marker means reconciliation will never find
it on restart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]