Mike and Wei,
Good news! I was able to manually live migrate these VMs following the steps
outlined below:
1.) virsh dumpxml 38 --migratable > 38.xml
2.) Change the vnc information in 38.xml to match destination host IP and
available VNC port
3.) virsh migrate --verbose --live 38 --xml 38.xml
qemu+tcp://destination.host.net/system
To my surprise, Cloudstack was able to discover and properly handle the fact
that this VM was live migrated to a new host without issue. Very cool.
Wei, I suspect you are correct when you said this was an issue with the
cloudstack agent code. After digging a little deeper, the agent is never
attempting to talk to libvirt at all after prepping the dxml to send to the
destination host. I'm going to attempt to reproduce this in my lab and attach
a remote debugger and see if I can get to the bottom of it.
Thanks again for the help guys! I really appreciate it.
Thanks,
David Mabry
On 1/30/18, 9:55 AM, "David Mabry" <dma...@ena.com.INVALID> wrote:
Ah, understood. I'll take a closer look at the logs and make sure that I
didn't accidentally miss those lines when I pulled together the logs for this
email chain.
Thanks,
David Mabry
On 1/30/18, 8:34 AM, "Wei ZHOU" <ustcweiz...@gmail.com> wrote:
Hi David,
I encountered the UnsupportAnswer once before, when I made some changes
in
the kvm plugin.
Normally there should be some network configurations in the agent.log
but I
do not see it.
-Wei
2018-01-30 15:00 GMT+01:00 David Mabry <dma...@ena.com.invalid>:
> Hi Wei,
>
> I detached the iso and received the same error. Just out of
curiosity,
> what leads you to believe it is something in the vxlan code? I guess
at
> this point, attaching a remote debugger to the agent in question
might be
> the best way to get to the bottom of what is going on.
>
> Thanks in advance for the help. I really, really appreciate it.
>
> Thanks,
> David Mabry
>
> On 1/30/18, 3:30 AM, "Wei ZHOU" <ustcweiz...@gmail.com> wrote:
>
> The answer should be caused by an exception in the cloudstack
agent.
> I tried to migrate a vm in our testing env, it is working.
>
> there are some different between our env and yours.
> (1) vlan VS vxlan
> (2) no ISO VS attached ISO
> (3) both of us use ceph and centos7.
>
> I suspect it is caused by codes on vxlan.
> However, could you detach the ISO and try again ?
>
> -Wei
>
>
>
> 2018-01-29 19:48 GMT+01:00 David Mabry <dma...@ena.com.invalid>:
>
> > Good day Cloudstack Devs,
> >
> > I've run across a real head scratcher. I have two VMs,
(initially 3
> VMs,
> > but more on that later) on a single host, that I cannot live
migrate
> to any
> > other host in the same cluster. We discovered this after
attempting
> to
> > roll out patches going from CentOS 7.2 to CentOS 7.4.
Initially, we
> > thought it had something to do with the new version of libvirtd
or
> qemu-kvm
> > on the other hosts in the cluster preventing these VMs from
> migrating, but
> > we are able to live migrate other VMs to and from this host
without
> issue.
> > We can even create new VMs on this specific host and live
migrate
> them
> > after creation with no issue. We've put the migration source
agent,
> > migration destination agent and the management server in debug
and
> don't
> > seem to get anything useful other than "Unsupported command".
> Luckily, we
> > did have one VM that was shutdown and restarted, this is the
3rd VM
> > mentioned above. Since that VM has been restarted, it has no
issues
> live
> > migrating to any other host in the cluster.
> >
> > I'm at a loss as to what to try next and I'm hoping that
someone out
> there
> > might have had a similar issue and could shed some light on
what to
> do.
> > Obviously, I can contact the customer and have them shutdown
their
> VMs, but
> > that will potentially just delay this problem to be solved
another
> day.
> > Even if shutting down the VMs is ultimately the solution, I'd
still
> like to
> > understand what happened to cause this issue in the first place
with
> the
> > hopes of preventing it in the future.
> >
> > Here's some information about my setup:
> > Cloudstack 4.8 Advanced Networking
> > CentOS 7.2 and 7.4 Hosts
> > Ceph RBD Primary Storage
> > NFS Secondary Storage
> > Instance in Question for Debug: i-532-1392-NSVLTN
> >
> > I have attached relevant debug logs to this email if anyone
wishes
> to take
> > a look. I think the most interesting error message that I have
> received is
> > the following:
> >
> > 468390:2018-01-27 08:59:35,172 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-6:ctx-188ea30f job-181792/job-181802
> ctx-8e7f45ad)
> > (logid:f0888362) Seq 22-942378222027276319: Received: { Ans: ,
> MgmtId:
> > 14038012703634, via: 22(csh02c01z01.nsvltn.ena.net), Ver: v1,
> Flags: 110,
> > { UnsupportedAnswer } }
> > 468391:2018-01-27 08:59:35,172 WARN [c.c.a.m.AgentManagerImpl]
> > (Work-Job-Executor-6:ctx-188ea30f job-181792/job-181802
> ctx-8e7f45ad)
> > (logid:f0888362) Unsupported Command: Unsupported command
issued:
> > com.cloud.agent.api.PrepareForMigrationCommand. Are you sure
you
> got the
> > right type of server?
> > 468392:2018-01-27 08:59:35,179 ERROR
[c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-6:ctx-188ea30f job-181792/job-181802
> ctx-8e7f45ad)
> > (logid:f0888362) Invocation exception, caused by:
> com.cloud.exception.AgentUnavailableException:
> > Resource [Host:22] is unreachable: Host 22: Unable to prepare
for
> migration
> > due to Unsupported command issued: com.cloud.agent.api.
> PrepareForMigrationCommand.
> > Are you sure you got the right type of server?
> > 468393:2018-01-27 08:59:35,179 INFO
[c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-6:ctx-188ea30f job-181792/job-181802
> ctx-8e7f45ad)
> > (logid:f0888362) Rethrow exception com.cloud.exception.
> AgentUnavailableException:
> > Resource [Host:22] is unreachable: Host 22: Unable to prepare
for
> migration
> > due to Unsupported command issued: com.cloud.agent.api.
> PrepareForMigrationCommand.
> > Are you sure you got the right type of server?
> >
> > I've tracked this "Unsupported command" down in the CS 4.8 code
to
> > cloudstack/api/src/com/cloud/agent/api/Answer.java which is the
> generic
> > answer class. I believe where the error is really being spawned
> from is
> > cloudstack/engine/orchestration/src/com/cloud/
> > vm/VirtualMachineManagerImpl.java. Specifically:
> > Answer pfma = null;
> > try {
> > pfma = _agentMgr.send(dstHostId, pfmc);
> > if (pfma == null || !pfma.getResult()) {
> > final String details = pfma != null ?
> pfma.getDetails() :
> > "null answer returned";
> > final String msg = "Unable to prepare for
migration
> due to
> > " + details;
> > pfma = null;
> > throw new AgentUnavailableException(msg,
dstHostId);
> > }
> >
> > The pfma returned must be in error or is never returned and
therefore
> > still null. That answer appears that it should be coming from
the
> > destination agent, but for the life of me I can't figure out
what
> the root
> > cause of this error is beyond, "Unsupported command issued".
What
> command
> > is unsupported? My guess is that it could be something wrong
with
> the dxml
> > that is generated and passed to the destination host, but I
have as
> yet
> > been unable to catch that dxml in debug.
> >
> > Any help or guidance is greatly appreciated.
> >
> > Thanks,
> > David Mabry
> >
> >
>
>
>
