Re: [Users] Problem with live migration.

Wed, 18 Apr 2012 02:06:28 -0700 

On 04/18/2012 11:30 AM, Martin Hovmöller wrote:




On Tue, Apr 17, 2012 at 4:38 PM, Rami Vaknin <[email protected] 
<mailto:[email protected]>> wrote:


    On 04/17/2012 05:36 PM, Martin Hovmöller wrote:



    On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <[email protected]
    <mailto:[email protected]>> wrote:

        On 04/17/2012 03:45 PM, Martin Hovmöller wrote:



        On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim
        <[email protected] <mailto:[email protected]>> wrote:

            On 04/17/2012 12:47 PM, Martin Hovmöller wrote:

                I can't live migrate the vm's in my cluster:

                012-04-17 04:06:39,728 INFO
                 [org.ovirt.engine.core.bll.MigrateVmCommand]
                (pool-5-thread-48)
                Running command: MigrateVmCommand internal: false.
                Entities affected :
                 ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM
                2012-04-17 04:06:39,743 INFO
                 [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
                (pool-5-thread-48)
                START, MigrateVDSCommand(vdsId =
                f33ffc14-87ba-11e1-b610-e3aeca1e8008,
                vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
                srcHost=10.23.30.130,
                dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,
                dstHost=10.23.30.110:54321
                <http://10.23.30.110:54321>
                <http://10.23.30.110:54321>,

                migrationMethod=ONLINE), log id: 7c7201be
                2012-04-17 04:06:39,748 INFO
                 
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
                (pool-5-thread-48) VdsBroker::migrate::Entered
                (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
                srcHost=10.23.30.130,
                dstHost=10.23.30.110:54321
                <http://10.23.30.110:54321>
                <http://10.23.30.110:54321>,  method=online

                2012-04-17 04:06:39,749 INFO
                 
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
                (pool-5-thread-48) START,
                MigrateBrokerVDSCommand(vdsId =
                f33ffc14-87ba-11e1-b610-e3aeca1e8008,
                vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a,
                srcHost=10.23.30.130,
                dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,
                dstHost=10.23.30.110:54321
                <http://10.23.30.110:54321>
                <http://10.23.30.110:54321>,

                migrationMethod=ONLINE), log id: 3c40c2de
                2012-04-17 04:06:39,834 INFO
                 
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
                (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand,
                log id: 3c40c2de
                2012-04-17 04:06:39,840 INFO
                 [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
                (pool-5-thread-48)
                FINISH, MigrateVDSCommand, return: MigratingFrom,
                log id: 7c7201be
                2012-04-17 04:06:42,166 INFO
                 [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
                (QuartzScheduler_Worker-18) vds::refreshVmList vm id
                ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to
                vds rhevh1.<domain>
                ignoring it in the refresh till migration is done
                [...]
                2012-04-17 04:07:33,852 INFO
                 [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
                (QuartzScheduler_Worker-34) vds::refreshVmList vm id
                ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused
                on vds
                rhevh1.<domain> ignoring it in the refresh till
                migration is done
                2012-04-17 04:07:37,407 ERROR
                [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
                (QuartzScheduler_Worker-71) Rerun vm
                ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from
                vds rhevh2.<domain>
                2012-04-17 04:07:37,414 INFO
                 
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
                (pool-5-thread-47) START,
                MigrateStatusVDSCommand(vdsId =
                f33ffc14-87ba-11e1-b610-e3aeca1e8008,
                vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id:
                8dfce1b
                2012-04-17 04:07:37,496 ERROR
                [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
                (pool-5-thread-47) Failed in MigrateStatusVDS method
                2012-04-17 04:07:37,497 ERROR
                [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
                (pool-5-thread-47) Error code createErr and error
                message
                VDSGenericException: VDSErrorException: Failed to
                MigrateStatusVDS,
                error = Error creating the requested virtual machine
                2012-04-17 04:07:37,497 INFO
                 [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
                (pool-5-thread-47) Command
                
org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand
                return
                value
                 Class Name:
                
org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
                mStatus                       Class Name:
                org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
                mCode                         9
                mMessage                      Error creating the
                requested virtual machine


                2012-04-17 04:07:37,497 INFO
                 [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
                (pool-5-thread-47) Vds: rhevh2.<domain>
                2012-04-17 04:07:37,497 ERROR
                [org.ovirt.engine.core.vdsbroker.VDSCommandBase]
                (pool-5-thread-47)
                Command MigrateStatusVDS execution failed.
                Exception: VDSErrorException:
                VDSGenericException: VDSErrorException: Failed to
                MigrateStatusVDS,
                error = Error creating the requested virtual machine
                2012-04-17 04:07:37,497 INFO
                 
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
                (pool-5-thread-47) FINISH, MigrateStatusVDSCommand,
                log id: 8dfce1b
                2012-04-17 04:07:37,528 WARN
                 [org.ovirt.engine.core.bll.MigrateVmCommand]
                (pool-5-thread-47)
                CanDoAction of action MigrateVm failed.
                
Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM


                There is no problem regarding storage or anything
                like that. If I shut
                down the vm I can start it on the other host without
                any problems
                whatsoever.
                Do I need to do something to make live migration
                work? Trying to dig in
                the logs on the hypervisors, but there's so much
                stuff being logged there...


                _______________________________________________
                Users mailing list
                [email protected] <mailto:[email protected]>
                http://lists.ovirt.org/mailman/listinfo/users


            vdsm log from the host that failed to create the VM?


        The vdsm log is frighteningly long, so I put it in pastebin:
        http://pastebin.com/BEeqFCCb
        I don't really see anything worthwhile myself, but on the
        other hand I'm not sure what I'm looking for, so it's mostly
        a massive wall of text to me, but it should be somewhere in
        that time frame.

        Here is the failure, "code 9" is not too informative, can you
        attach the libvirtd.log file too?

        Thread-49287::DEBUG::2012-04-17
        14:25:39,834::clientIF::81::vds::(wrapper) return
        migrationCreate with {'status': {'message': 'Error creating
        the requested virtual machine', 'code': 9}}




        _______________________________________________
        Users mailing list
        [email protected]  <mailto:[email protected]>
        http://lists.ovirt.org/mailman/listinfo/users




        -- 


        Thanks,

        Rami Vaknin, QE @ Red Hat, TLV, IL.


    I think this is the relevant info from the libvirt.log:
    2012-04-17 14:30:53.768+0000: 8506: debug :
    virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0,
    uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
    2012-04-17 14:30:53.768+0000: 8506: debug :
    virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0,
    uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a
    2012-04-17 14:30:53.771+0000: 8506: error :
    virNetClientProgramDispatchError:170 : Domain not found: no
    domain with matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'

    What domain is this, is it the storage domain?


    Domain in libvirt is simply a vm.




    _______________________________________________
    Users mailing list
    [email protected]  <mailto:[email protected]>
    http://lists.ovirt.org/mailman/listinfo/users




    -- 


    Thanks,

    Rami Vaknin, QE @ Red Hat, TLV, IL.



Alright, so what does this mean then? The machine I'm migrating to 
can't see the vm?

That doesn't make sense since I can manually start it on that host..



On second look at the vdsm.log I see that disk preparation in the 
destination host failed (timed out), try to take a look at vdsm.log at 
the destination.
In addition, for some reason getVmStats returns "Virtual machine does 
not exist" right before the migration.




Thread-49286::DEBUG::2012-04-17 
14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call 
getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}
Thread-49286::DEBUG::2012-04-17 
14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with 
{'status': {'message': 'Virtual machine does not exist', 'code': 1}}
Thread-49287::DEBUG::2012-04-17 
14:25:07,381::clientIF::76::vds::(wrapper) [10.23.30.130]::call 
migrationCreate with ({'bridge': 'ovirtmgmt', 'acpiEnable': 'true',.....
Thread-49287::DEBUG::2012-04-17 
14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare) 
vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration destination: 
waiting 31s for path preparation



Thread-49287::DEBUG::2012-04-17 
14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare) 
vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while waiting for 
path preparation



14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call 
getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}
Thread-49286::DEBUG::2012-04-17 
14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with 
{'status': {'message': 'Virtual machine does not exist', 'code': 1}}



_______________________________________________
Users mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/users



--

Thanks,

Rami Vaknin, QE @ Red Hat, TLV, IL.


_______________________________________________
Users mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/users






















					Reply via email to