ccycv opened a new issue, #6270:
URL: https://github.com/apache/cloudstack/issues/6270
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete
the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
Storage
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main
branch.
-->
~~~
4.15.x to 4.16.1
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network,
advanced networking, etc. N/A otherwise
-->
~~~
ACS 4.16.x with VMware 6.5 - Advanced Networking - Local Storage
~~~
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
~~~
CentOS 7.9, Ubuntu 20
~~~
##### SUMMARY
<!-- Explain the problem/feature briefly -->
The migration for VMs with local storage is not working, tested in 3
different environment with similar setups. I verified the VMware for
cloud.uuid, it is the same uuid as in the cloudstack.
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal
test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
- Stop Instance
- Migrate instance to another primary storage
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
Migrate the volume with success, in a older version was working
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
2022-04-15 06:18:52,030 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-309:ctx-515baff0 fra-de-001.domain.tld, job-2857/job-2858, cmd:
MigrateVmToPoolCommand) (logid:e8851491) Unable to find the target datastore:
70f486ac-f952-463b-90c7-3e3dc2843b62 in datacenter: EMEA-DE-01 to execute
migration
2022-04-15 06:18:52,080 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-309:ctx-515baff0 fra-de-001.domain.tld, job-2857/job-2858, cmd:
MigrateVmToPoolCommand) (logid:e8851491) Change data store for VM i-2-546-VM
failed: Unable to find the target datastore:
70f486ac-f952-463b-90c7-3e3dc2843b62 in datacenter: EMEA-DE-01 to execute
migration
2022-04-15 06:18:52,086 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-309:ctx-515baff0) (logid:e8851491) Seq 1-3247939756264335318:
Response Received:
2022-04-15 06:18:52,088 DEBUG [c.c.a.t.Request]
(DirectAgent-309:ctx-515baff0) (logid:e8851491) Seq 1-3247939756264335318:
Processing: { Ans: , MgmtId: 39908814623087, via: 1(fra-de-001.domain.tld),
Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"Exception:
com.cloud.utils.exception.CloudRuntimeException
Message: Change data store for VM i-2-546-VM failed
Stack: com.cloud.utils.exception.CloudRuntimeException: Change data store
for VM i-2-546-VM failed
at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareResource.java:4550)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:4507)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:501)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to find
the target datastore: 70f486ac-f952-463b-90c7-3e3dc2843b62 in datacenter:
EMEA-DE-01 to execute migration
at
com.cloud.hypervisor.vmware.resource.VmwareResource.relocateVirtualMachine(VmwareResource.java:7588)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareResource.java:4545)
... 14 more
","wait":"0","bypassHostMaintenance":"false"}}] }
2022-04-15 06:18:52,088 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-309:ctx-515baff0) (logid:e8851491) Seq 1-3247939756264335318: No
more commands found
2022-04-15 06:18:52,088 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858 ctx-8da49044)
(logid:e8851491) Seq 1-3247939756264335318: Received: { Ans: , MgmtId:
39908814623087, via: 1(fra-de-001.domain.tld), Ver: v1, Flags: 110, { Answer } }
2022-04-15 06:18:52,088 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858 ctx-8da49044)
(logid:e8851491) Cleaning up after hypervisor pool migration volumes for VM
i-2-546-VM(143774cd-6cd7-4064-8383-45cf30c55afc)
2022-04-15 06:18:52,097 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858 ctx-8da49044)
(logid:e8851491) VM instance {id: "546", name: "i-2-546-VM", uuid:
"143774cd-6cd7-4064-8383-45cf30c55afc", type="User"} state transited from
[Migrating] to [Stopped] with event [AgentReportStopped]. VM's original host:
Host {"id": "1", "name": "fra-de-001.domain.tld", "uuid":
"d9b7391b-4ff6-4e0a-bcc2-697e472801c4", "type"="Routing"}, new host: null, host
before state transition: null
2022-04-15 06:18:52,100 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858 ctx-8da49044)
(logid:e8851491) Hosts's actual total CPU: 76640 and CPU after applying
overprovisioning: 229920
2022-04-15 06:18:52,100 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858 ctx-8da49044)
(logid:e8851491) Hosts's actual total RAM: (127.89 GB) 137320136704 and RAM
after applying overprovisioning: (255.78 GB) 274640273408
2022-04-15 06:18:52,100 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858 ctx-8da49044)
(logid:e8851491) release cpu from host: 1, old used: 162600,reserved: 2000,
actual total: 76640, total with overprovisioning: 229920; new used:
160600,reserved:2000; movedfromreserved: false,moveToReserveredfalse
2022-04-15 06:18:52,100 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858 ctx-8da49044)
(logid:e8851491) release mem from host: 1, old used: (170.75 GB)
183341416448,reserved: (2.00 GB) 2147483648, total: (255.78 GB) 274640273408;
new used: (168.75 GB) 181193932800,reserved:(2.00 GB) 2147483648;
movedfromreserved: false,moveToReserveredfalse
2022-04-15 06:18:52,107 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858 ctx-8da49044)
(logid:e8851491) Invocation exception, caused by:
com.cloud.utils.exception.CloudRuntimeException: VM ID:
143774cd-6cd7-4064-8383-45cf30c55afc migration failed. Exception:
com.cloud.utils.exception.CloudRuntimeException
Message: Change data store for VM i-2-546-VM failed
Stack: com.cloud.utils.exception.CloudRuntimeException: Change data store
for VM i-2-546-VM failed
at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareResource.java:4550)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:4507)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:501)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to find
the target datastore: 70f486ac-f952-463b-90c7-3e3dc2843b62 in datacenter:
EMEA-DE-01 to execute migration
at
com.cloud.hypervisor.vmware.resource.VmwareResource.relocateVirtualMachine(VmwareResource.java:7588)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareResource.java:4545)
... 14 more
2022-04-15 06:18:52,107 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858 ctx-8da49044)
(logid:e8851491) Rethrow exception
com.cloud.utils.exception.CloudRuntimeException: VM ID:
143774cd-6cd7-4064-8383-45cf30c55afc migration failed. Exception:
com.cloud.utils.exception.CloudRuntimeException
Message: Change data store for VM i-2-546-VM failed
Stack: com.cloud.utils.exception.CloudRuntimeException: Change data store
for VM i-2-546-VM failed
at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareResource.java:4550)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:4507)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:501)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to find
the target datastore: 70f486ac-f952-463b-90c7-3e3dc2843b62 in datacenter:
EMEA-DE-01 to execute migration
at
com.cloud.hypervisor.vmware.resource.VmwareResource.relocateVirtualMachine(VmwareResource.java:7588)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareResource.java:4545)
... 14 more
2022-04-15 06:18:52,107 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858) (logid:e8851491) Done
with run of VM work job: com.cloud.vm.VmWorkStorageMigration for VM 546, job
origin: 2857
2022-04-15 06:18:52,107 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858) (logid:e8851491) Unable
to complete AsyncJobVO: {id:2858, userId: 2, accountId: 2, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStorageMigration, cmdInfo:
rO0ABXNyACNjb20uY2xvdWQudm0uVm1Xb3JrU3RvcmFnZU1pZ3JhdGlvboeRn3LBtueeAgABTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdXRpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAIidAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAx3CAAAABAAAAABc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAInc3EAfgAIAAAAAAAAAAp4,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 39908814623087, completeMsid: null, lastUpdated: null,
lastPolled: null
, created: Fri Apr 15 06:18:48 UTC 2022, removed: null}, job origin:2857
com.cloud.utils.exception.CloudRuntimeException: VM ID:
143774cd-6cd7-4064-8383-45cf30c55afc migration failed. Exception:
com.cloud.utils.exception.CloudRuntimeException
Message: Change data store for VM i-2-546-VM failed
Stack: com.cloud.utils.exception.CloudRuntimeException: Change data store
for VM i-2-546-VM failed
at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareResource.java:4550)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:4507)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:501)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to find
the target datastore: 70f486ac-f952-463b-90c7-3e3dc2843b62 in datacenter:
EMEA-DE-01 to execute migration
at
com.cloud.hypervisor.vmware.resource.VmwareResource.relocateVirtualMachine(VmwareResource.java:7588)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareResource.java:4545)
... 14 more
at
com.cloud.vm.VirtualMachineManagerImpl.markVolumesInPool(VirtualMachineManagerImpl.java:2403)
at
com.cloud.vm.VirtualMachineManagerImpl.afterHypervisorMigrationCleanup(VirtualMachineManagerImpl.java:2395)
at
com.cloud.vm.VirtualMachineManagerImpl.migrateThroughHypervisorOrStorage(VirtualMachineManagerImpl.java:2459)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStorageMigration(VirtualMachineManagerImpl.java:2325)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStorageMigration(VirtualMachineManagerImpl.java:6011)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6018)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-04-15 06:18:52,116 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858) (logid:e8851491)
Complete async job-2858, jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0CEJWTSBJRDogMTQzNzc0Y2QtNmNkNy00MDY0LTgzODMtNDVjZjMwYzU1YWZjIG1pZ3JhdGlvbiBmYWlsZWQuIEV4Y2VwdGlvbjogY29tLmNsb3VkLnV0aWxzLmV4Y2VwdGlvbi5DbG91ZFJ1bnRpbWVFeGNlcHRpb24KTWVzc2FnZTogQ2hhbmdlIGRhdGEgc3RvcmUgZm9yIFZNIGktMi01NDYtVk0gZmFpbGVkClN0YWNrOiBjb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbjogQ2hhbmdlIGRh
dGEgc3RvcmUgZm9yIFZNIGktMi01NDYtVk0gZmFpbGVkCglhdCBjb20uY2xvdWQuaHlwZXJ2aXNvci52bXdhcmUucmVzb3VyY2UuVm13YXJlUmVzb3VyY2UubWlncmF0ZUFuZEFuc3dlcihWbXdhcmVSZXNvdXJjZS5qYXZhOjQ1NTApCglhdCBjb20uY2xvdWQuaHlwZXJ2aXNvci52bXdhcmUucmVzb3VyY2UuVm13YXJlUmVzb3VyY2UuZXhlY3V0ZShWbXdhcmVSZXNvdXJjZS5qYXZhOjQ1MDcpCglhdCBjb20uY2xvdWQuaHlwZXJ2aXNvci52bXdhcmUucmVzb3VyY2UuVm13YXJlUmVzb3VyY2UuZXhlY3V0ZVJlcXVlc3QoVm13YXJlUmVzb3VyY2UuamF2YTo1MDEpCglhdCBjb20uY2xvdWQuYWdlbnQubWFuYWdlci5EaXJlY3RBZ2VudEF0dGFjaGUkVGFzay5ydW5JbkNvbnRleHQoRGlyZWN0QWdlbnRBdHRhY2hlLmphdmE6MzE1KQoJYXQgb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDEucnVuKE1hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YTo0OCkKCWF0IG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMS5jYWxsKERlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhOjU1KQoJYXQgb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5jYWxsV2l0aENvbnRleHQoRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmE6MTAyKQoJYXQgb3JnL
mFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5ydW5XaXRoQ29udGV4dChEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YTo1MikKCWF0IG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZS5ydW4oTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhOjQ1KQoJYXQgamF2YS5iYXNlL2phdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXIuY2FsbChFeGVjdXRvcnMuamF2YTo1MTUpCglhdCBqYXZhLmJhc2UvamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFzay5ydW4oRnV0dXJlVGFzay5qYXZhOjI2NCkKCWF0IGphdmEuYmFzZS9qYXZhLnV0aWwuY29uY3VycmVudC5TY2hlZHVsZWRUaHJlYWRQb29sRXhlY3V0b3IkU2NoZWR1bGVkRnV0dXJlVGFzay5ydW4oU2NoZWR1bGVkVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmE6MzA0KQoJYXQgamF2YS5iYXNlL2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvci5ydW5Xb3JrZXIoVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmE6MTEyOCkKCWF0IGphdmEuYmFzZS9qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VyLnJ1bihUaHJlYWRQb29sRXhlY3V0b3IuamF2YTo2MjgpCglhdCBqYXZhLmJhc2UvamF2YS5sYW5nLlRocmVhZC5ydW4oVGhyZWFkLmphdmE6ODI5KQpDYXVzZWQgYnk6IGNvbS5jbG
91ZC51dGlscy5leGNlcHRpb24uQ2xvdWRSdW50aW1lRXhjZXB0aW9uOiBVbmFibGUgdG8gZmluZCB0aGUgdGFyZ2V0IGRhdGFzdG9yZTogNzBmNDg2YWMtZjk1Mi00NjNiLTkwYzctM2UzZGMyODQzYjYyIGluIGRhdGFjZW50ZXI6IEVNRUEtREUtMDEgdG8gZXhlY3V0ZSBtaWdyYXRpb24KCWF0IGNvbS5jbG91ZC5oeXBlcnZpc29yLnZtd2FyZS5yZXNvdXJjZS5WbXdhcmVSZXNvdXJjZS5yZWxvY2F0ZVZpcnR1YWxNYWNoaW5lKFZtd2FyZVJlc291cmNlLmphdmE6NzU4OCkKCWF0IGNvbS5jbG91ZC5oeXBlcnZpc29yLnZtd2FyZS5yZXNvdXJjZS5WbXdhcmVSZXNvdXJjZS5taWdyYXRlQW5kQW5zd2VyKFZtd2FyZVJlc291cmNlLmphdmE6NDU0NSkKCS4uLiAxNCBtb3JlCnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAGHNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAJY3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAARbWFya1ZvbHVtZXNJblBvb2xwcHNxAH4ADAEAAAlbcQB-AA5xAH4AD3EAfgA
QdAAfYWZ0ZXJIeXBlcnZpc29yTWlncmF0aW9uQ2xlYW51cHBwc3EAfgAMAQAACZtxAH4ADnEAfgAPcQB-ABB0ACFtaWdyYXRlVGhyb3VnaEh5cGVydmlzb3JPclN0b3JhZ2VwcHNxAH4ADAEAAAkVcQB-AA5xAH4AD3EAfgAQdAAbb3JjaGVzdHJhdGVTdG9yYWdlTWlncmF0aW9ucHBzcQB-AAwBAAAXe3EAfgAOcQB-AA9xAH4AEHEAfgAXcHBzcQB-AAwC_____nB0AC1qZGsuaW50ZXJuYWwucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2Nlc3NvckltcGwuamF2YXQAB2ludm9rZTB0AAlqYXZhLmJhc2V0AAkxMS4wLjE0LjFzcQB-AAwCAAAAPnBxAH4AGnEAfgAbdAAGaW52b2tlcQB-AB1xAH4AHnNxAH4ADAIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsLmphdmFxAH4AIHEAfgAdcQB-AB5zcQB-AAwCAAACNnB0ABhqYXZhLmxhbmcucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAgcQB-AB1xAH4AHnNxAH4ADAEAAABrcQB-AA50ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4ADAEAABeCcQB-AA5xAH4AD3EAfgAQcQB-ACpwcHNxAH4ADAEAAABmcQB-AA50ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNo
ZXIuamF2YXQABnJ1bkpvYnBwc3EAfgAMAQAAAmxxAH4ADnQAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHBwc3EAfgAMAQAAADBxAH4ADnQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVucHBzcQB-AAwBAAAAN3EAfgAOdABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxscHBzcQB-AAwBAAAAZnEAfgAOdABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA6dAAPY2FsbFdpdGhDb250ZXh0cHBzcQB-AAwBAAAANHEAfgAOcQB-AD1xAH4AOnQADnJ1bldpdGhDb250ZXh0cHBzcQB-AAwBAAAALXEAfgAOdAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-ADZxAH4AN3Bwc3EAfgAMAQAAAjhxAH4ADnEAfgAxcQB-ADJxAH4AN3Bwc3EAfgAMAgAAAgNwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-ADtxA
H4AHXEAfgAec3EAfgAMAgAAAQhwdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgA3cQB-AB1xAH4AHnNxAH4ADAIAAARocHQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2VycQB-AB1xAH4AHnNxAH4ADAIAAAJ0cHQALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4ATHEAfgA3cQB-AB1xAH4AHnNxAH4ADAIAAAM9cHQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgA3cQB-AB1xAH4AHnNyAB9qYXZhLnV0aWwuQ29sbGVjdGlvbnMkRW1wdHlMaXN0ergXtDynnt4CAAB4cHgAABCadwgAAAAAAAAAAHg
2022-04-15 06:18:52,117 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858) (logid:e8851491) Publish
async job-2858 complete on message bus
2022-04-15 06:18:52,117 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858) (logid:e8851491) Wake up
jobs related to job-2858
2022-04-15 06:18:52,117 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858) (logid:e8851491) Update
db status for job-2858
2022-04-15 06:18:52,117 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858) (logid:e8851491) Wake up
jobs joined with job-2858 and disjoin all subjobs created from job- 2858
2022-04-15 06:18:52,121 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858) (logid:e8851491) Done
executing com.cloud.vm.VmWorkStorageMigration for job-2858
2022-04-15 06:18:52,121 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-120:ctx-4c5f6d33 job-2857/job-2858) (logid:e8851491) Remove
job-2858 from job monitoring
2022-04-15 06:18:52,125 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-101:ctx-26e9f7c1 job-2857) (logid:e8851491) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
com.cloud.utils.exception.CloudRuntimeException: VM ID:
143774cd-6cd7-4064-8383-45cf30c55afc migration failed. Exception:
com.cloud.utils.exception.CloudRuntimeException
Message: Change data store for VM i-2-546-VM failed
Stack: com.cloud.utils.exception.CloudRuntimeException: Change data store
for VM i-2-546-VM failed
at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareResource.java:4550)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:4507)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:501)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to find
the target datastore: 70f486ac-f952-463b-90c7-3e3dc2843b62 in datacenter:
EMEA-DE-01 to execute migration
at
com.cloud.hypervisor.vmware.resource.VmwareResource.relocateVirtualMachine(VmwareResource.java:7588)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareResource.java:4545)
... 14 more
at
com.cloud.vm.VirtualMachineManagerImpl.markVolumesInPool(VirtualMachineManagerImpl.java:2403)
at
com.cloud.vm.VirtualMachineManagerImpl.afterHypervisorMigrationCleanup(VirtualMachineManagerImpl.java:2395)
at
com.cloud.vm.VirtualMachineManagerImpl.migrateThroughHypervisorOrStorage(VirtualMachineManagerImpl.java:2459)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStorageMigration(VirtualMachineManagerImpl.java:2325)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStorageMigration(VirtualMachineManagerImpl.java:6011)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6018)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-04-15 06:18:52,127 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-101:ctx-26e9f7c1 job-2857) (logid:e8851491) Complete async
job-2857, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"VM
ID: 143774cd-6cd7-4064-8383-45cf30c55afc migration failed. Exception:
com.cloud.utils.exception.CloudRuntimeException
Message: Change data store for VM i-2-546-VM failed
Stack: com.cloud.utils.exception.CloudRuntimeException: Change data store
for VM i-2-546-VM failed
at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareResource.java:4550)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:4507)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:501)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to find
the target datastore: 70f486ac-f952-463b-90c7-3e3dc2843b62 in datacenter:
EMEA-DE-01 to execute migration
at
com.cloud.hypervisor.vmware.resource.VmwareResource.relocateVirtualMachine(VmwareResource.java:7588)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.migrateAndAnswer(VmwareResource.java:4545)
... 14 more
"}
2022-04-15 06:18:52,127 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-101:ctx-26e9f7c1 job-2857) (logid:e8851491) Publish async
job-2857 complete on message bus
2022-04-15 06:18:52,127 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-101:ctx-26e9f7c1 job-2857) (logid:e8851491) Wake up jobs
related to job-2857
2022-04-15 06:18:52,127 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-101:ctx-26e9f7c1 job-2857) (logid:e8851491) Update db status
for job-2857
2022-04-15 06:18:52,128 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-101:ctx-26e9f7c1 job-2857) (logid:e8851491) Wake up jobs
joined with job-2857 and disjoin all subjobs created from job- 2857
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
