Hi David,
So, I don’t know if what I am going to say here will at all be of use to you,
but maybe. :)
I had a customer one time mention to me that he had trouble with live VM
migration on KVM with a VM that was created on an older version of CloudStack.
Live VM migration worked fine for these VMs on the older version of CloudStack
(I think it was version 4.5) and stopped working when he upgraded to 4.8. New
VMs (VMs created on the newer version of CloudStack) worked fine for this
feature on 4.8, but old VMs had to be stopped and re-started for live VM
migration to work. I believe the older version of CloudStack was not placing
the serial number of the VM in the VM’s XML descriptor file, but newer versions
of CloudStack were expecting this field.
Can you dump the XML of one or both of your VMs that don’t live migrate and see
if they have the serial number field in their XML? Then, I’d recommend dumping
the XML of the VM that works and seeing if it does, in fact, have the serial
number field in its XML.
I hope this is of some help.
Talk to you later,
Mike
On 1/29/18, 11:48 AM, "David Mabry" <dma...@ena.com.INVALID> wrote:
Good day Cloudstack Devs,
I've run across a real head scratcher. I have two VMs, (initially 3 VMs,
but more on that later) on a single host, that I cannot live migrate to any
other host in the same cluster. We discovered this after attempting to roll
out patches going from CentOS 7.2 to CentOS 7.4. Initially, we thought it had
something to do with the new version of libvirtd or qemu-kvm on the other hosts
in the cluster preventing these VMs from migrating, but we are able to live
migrate other VMs to and from this host without issue. We can even create new
VMs on this specific host and live migrate them after creation with no issue.
We've put the migration source agent, migration destination agent and the
management server in debug and don't seem to get anything useful other than
"Unsupported command". Luckily, we did have one VM that was shutdown and
restarted, this is the 3rd VM mentioned above. Since that VM has been
restarted, it has no issues live migrating to any other host in the cluster.
I'm at a loss as to what to try next and I'm hoping that someone out there
might have had a similar issue and could shed some light on what to do.
Obviously, I can contact the customer and have them shutdown their VMs, but
that will potentially just delay this problem to be solved another day. Even
if shutting down the VMs is ultimately the solution, I'd still like to
understand what happened to cause this issue in the first place with the hopes
of preventing it in the future.
Here's some information about my setup:
Cloudstack 4.8 Advanced Networking
CentOS 7.2 and 7.4 Hosts
Ceph RBD Primary Storage
NFS Secondary Storage
Instance in Question for Debug: i-532-1392-NSVLTN
I have attached relevant debug logs to this email if anyone wishes to take
a look. I think the most interesting error message that I have received is the
following:
468390:2018-01-27 08:59:35,172 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-6:ctx-188ea30f job-181792/job-181802 ctx-8e7f45ad)
(logid:f0888362) Seq 22-942378222027276319: Received: { Ans: , MgmtId:
14038012703634, via: 22(csh02c01z01.nsvltn.ena.net), Ver: v1, Flags: 110, {
UnsupportedAnswer } }
468391:2018-01-27 08:59:35,172 WARN [c.c.a.m.AgentManagerImpl]
(Work-Job-Executor-6:ctx-188ea30f job-181792/job-181802 ctx-8e7f45ad)
(logid:f0888362) Unsupported Command: Unsupported command issued:
com.cloud.agent.api.PrepareForMigrationCommand. Are you sure you got the right
type of server?
468392:2018-01-27 08:59:35,179 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-6:ctx-188ea30f job-181792/job-181802 ctx-8e7f45ad)
(logid:f0888362) Invocation exception, caused by:
com.cloud.exception.AgentUnavailableException: Resource [Host:22] is
unreachable: Host 22: Unable to prepare for migration due to Unsupported
command issued: com.cloud.agent.api.PrepareForMigrationCommand. Are you sure
you got the right type of server?
468393:2018-01-27 08:59:35,179 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-6:ctx-188ea30f job-181792/job-181802 ctx-8e7f45ad)
(logid:f0888362) Rethrow exception
com.cloud.exception.AgentUnavailableException: Resource [Host:22] is
unreachable: Host 22: Unable to prepare for migration due to Unsupported
command issued: com.cloud.agent.api.PrepareForMigrationCommand. Are you sure
you got the right type of server?
I've tracked this "Unsupported command" down in the CS 4.8 code to
cloudstack/api/src/com/cloud/agent/api/Answer.java which is the generic answer
class. I believe where the error is really being spawned from is
cloudstack/engine/orchestration/src/com/cloud/vm/VirtualMachineManagerImpl.java.
Specifically:
Answer pfma = null;
try {
pfma = _agentMgr.send(dstHostId, pfmc);
if (pfma == null || !pfma.getResult()) {
final String details = pfma != null ? pfma.getDetails() :
"null answer returned";
final String msg = "Unable to prepare for migration due to
" + details;
pfma = null;
throw new AgentUnavailableException(msg, dstHostId);
}
The pfma returned must be in error or is never returned and therefore still
null. That answer appears that it should be coming from the destination agent,
but for the life of me I can't figure out what the root cause of this error is
beyond, "Unsupported command issued". What command is unsupported? My guess
is that it could be something wrong with the dxml that is generated and passed
to the destination host, but I have as yet been unable to catch that dxml in
debug.
Any help or guidance is greatly appreciated.
Thanks,
David Mabry
