Sure thing ----- Original Message ----- From: "Dafna Ron" <[email protected]> To: "Maurice James" <[email protected]> Cc: "Liron Aravot" <[email protected]>, "itamar Heim" <[email protected]>, "elad Ben Aharon" <[email protected]>, [email protected] Sent: Thursday, April 17, 2014 6:50:44 AM Subject: Re: [ovirt-users] Error creating Disks
valid and important discussion indeed. important enough I think that this should be raised in a separate thread allowing others to discuss it (since this is an on-going issue some people may not be aware of a subject change). Can you please send a new mail with a relevant headline raising this issue? Thanks, Dafna On 04/17/2014 11:47 AM, Maurice James wrote: > Just curious. I have come across a few major problems since upgrading to > 3.4.x (External Authentication BZ1081204, Quota assignment BZ1081014, Live > migration "qemu bug" ) to name a few. I imagine that all of the features need > to be polished before inclusion into RHEL. I'm trying to get my company to > adopt RHEL instead of V-Sphere but that is becoming a little tricky. Minus > the cool new features of 3.4.x 3.3.4 was pretty stable IMO. > > ----- Original Message ----- > From: "Dafna Ron" <[email protected]> > To: "Maurice James" <[email protected]>, "Liron Aravot" > <[email protected]>, "itamar Heim" <[email protected]> > Cc: "elad Ben Aharon" <[email protected]>, [email protected] > Sent: Thursday, April 17, 2014 5:31:46 AM > Subject: Re: [ovirt-users] Error creating Disks > > I am not sure this question relates to this thread and perhaps it should > be posted in a different one :) can you explain what you mean by that > question? > > > On 04/17/2014 01:09 AM, Maurice James wrote: >> Which version of Ovirt are you guys going to build the new RHEV from? >> >> ----- Original Message ----- >> From: "Dafna Ron" <[email protected]> >> To: "Maurice James" <[email protected]> >> Cc: "elad Ben Aharon" <[email protected]>, "Liron Aravot" >> <[email protected]>, [email protected] >> Sent: Wednesday, April 16, 2014 5:39:20 PM >> Subject: Re: [ovirt-users] Error creating Disks >> >> Ok. >> so it's the qemu bug that Liron sent and I think there is some bug there >> with the engine cache since we did not see the job failing in vdsm log. >> hopefully there will be a qemu patch for centos soon... >> >> Thanks Maurice! >> >> Dafna >> >> >> On 04/16/2014 07:14 PM, Maurice James wrote: >>> The offline disk migration works >>> >>> ----- Original Message ----- >>> From: "Dafna Ron" <[email protected]> >>> To: "Maurice James" <[email protected]> >>> Cc: "elad Ben Aharon" <[email protected]>, "Liron Aravot" >>> <[email protected]>, [email protected] >>> Sent: Wednesday, April 16, 2014 11:57:20 AM >>> Subject: Re: [ovirt-users] Error creating Disks >>> >>> Thanks Maurice. >>> so you are saying that there is some sort of caching. >>> >>> can you migrate the disk off line? >>> >>> Dafna >>> >>> >>> On 04/16/2014 04:06 PM, Maurice James wrote: >>>> I ran "tail -f /var/log/vdsm/vdsm.log |grep ERROR" while attempting a live >>>> migration and nothing is coming up, but >>>> "tail -f /var/log/ovirt-engine/engine.log |grep ERROR" returns: >>>> 2014-04-16 11:02:59,564 ERROR >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] >>>> (org.ovirt.thread.pool-6-thread-40) Failed in SnapshotVDS method >>>> 2014-04-16 11:02:59,568 ERROR >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] >>>> (org.ovirt.thread.pool-6-thread-40) Command SnapshotVDSCommand(HostName = >>>> vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, >>>> vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. Exception: >>>> VDSErrorException: VDSGenericException: VDSErrorException: Failed to >>>> SnapshotVDS, error = Snapshot failed, code = 48 >>>> 2014-04-16 11:02:59,966 ERROR >>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>> (org.ovirt.thread.pool-6-thread-40) Command >>>> org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll >>>> exception. With error message VdcBLLException: Auto-generated live >>>> snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed with >>>> error imageErr and code 13) >>>> 2014-04-16 11:02:59,970 ERROR >>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>> (org.ovirt.thread.pool-6-thread-40) Reverting task unknown, handler: >>>> org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler >>>> >>>> >>>> >>>> ----- Original Message ----- >>>> From: "Dafna Ron" <[email protected]> >>>> To: "Maurice James" <[email protected]>, "elad Ben Aharon" >>>> <[email protected]> >>>> Cc: "Liron Aravot" <[email protected]>, [email protected] >>>> Sent: Wednesday, April 16, 2014 11:00:47 AM >>>> Subject: Re: [ovirt-users] Error creating Disks >>>> >>>> Since LiveStorageMigration is a complicated and long task which divides >>>> into 4 different jobs I asked for the log with the echo before the task >>>> so we can easily follow the task from start to end. >>>> >>>> however, I did grep for the task id in the logs you just attached and >>>> there is nothing there. >>>> I'm adding Elad to try and see if he can reproduce ERROR populated by >>>> engine because of cache. >>>> >>>> Dafna >>>> >>>> >>>> On 04/16/2014 03:45 PM, Maurice James wrote: >>>>> I attached a few of the rotated logs, something might be in there >>>>> >>>>> ----- Original Message ----- >>>>> From: "Dafna Ron" <[email protected]> >>>>> To: "Liron Aravot" <[email protected]> >>>>> Cc: "Maurice James" <[email protected]>, [email protected] >>>>> Sent: Wednesday, April 16, 2014 10:41:59 AM >>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>> >>>>> Thanks Liron, >>>>> >>>>> well, unless the first vdsm log was cut we were not seeing any errors in >>>>> the vdsm log (engine was reporting an issue and vdsm did not show any >>>>> errors at all), after the engine restart we can see an error in the vdsm >>>>> log. >>>>> Maurice, was is possible that the vdsm log was cut? if not, there might >>>>> be a second bug with engine cache. >>>>> >>>>> The Error we are seeing in the vdsm log now does indeed look like the >>>>> live snapshot issue in qemu :) >>>>> >>>>> >>>>> Thanks, >>>>> Dafna >>>>> >>>>> >>>>> >>>>> On 04/16/2014 03:23 PM, Liron Aravot wrote: >>>>>> Hi Maurice, Dafna >>>>>> The creation of the snapshot for each of the disks succeeds, but >>>>>> performing the "live" part of it fails - we can see the following error >>>>>> in the vdsm log. >>>>>> >>>>>> Thread-4444::DEBUG::2014-04-16 09:40:39,071::vm::4007::vm.Vm::(snapshot) >>>>>> vmId=`50cf8bce-3982-491a-8b67-7d009c5c3243`::Snapshot faile >>>>>> d using the quiesce flag, trying again without it (unsupported >>>>>> configuration: reuse is not supported with this QEMU binary) >>>>>> Thread-4444::DEBUG::2014-04-16 >>>>>> 09:40:39,081::libvirtconnection::124::root::(wrapper) Unknown >>>>>> libvirterror: ecode: 67 edom: 10 level: >>>>>> 2 message: unsupported configuration: reuse is not supported with >>>>>> this QEMU binary >>>>>> Thread-4444::ERROR::2014-04-16 09:40:39,082::vm::4011::vm.Vm::(snapshot) >>>>>> vmId=`50cf8bce-3982-491a-8b67-7d009c5c3243`::Unable to take >>>>>> snapshot >>>>>> Traceback (most recent call last): >>>>>> File "/usr/share/vdsm/vm.py", line 4009, in snapshot >>>>>> self._dom.snapshotCreateXML(snapxml, snapFlags) >>>>>> File "/usr/share/vdsm/vm.py", line 859, in f >>>>>> ret = attr(*args, **kwargs) >>>>>> File >>>>>> "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 92, >>>>>> in wrapper >>>>>> ret = f(*args, **kwargs) >>>>>> File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1636, >>>>>> in snapshotCreateXML >>>>>> if ret is None:raise >>>>>> libvirtError('virDomainSnapshotCreateXML() failed', dom=self) >>>>>> libvirtError: unsupported configuration: reuse is not supported with >>>>>> this QEMU binary >>>>>> Thread-4444::DEBUG::2014-04-16 >>>>>> 09:40:39,091::BindingXMLRPC::1074::vds::(wrapper) return vmSnapshot with >>>>>> {'status': {'message': 'Snap >>>>>> shot failed', 'code': 48}} >>>>>> >>>>>> there's open bug on possibly the same issue - but let's verify that it's >>>>>> the same >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1009100 >>>>>> >>>>>> what OS are you running? if it's not centos, please try to upgrade >>>>>> libvirt and try again. >>>>>> >>>>>> if it's urgent - as i see that you already stopped your vm during those >>>>>> tries, as a temporary solution you can stop the vm, move the disk while >>>>>> it's stopped (which won't be live storage migration) and than start it. >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Dafna Ron" <[email protected]> >>>>>>> To: "Maurice James" <[email protected]> >>>>>>> Cc: [email protected] >>>>>>> Sent: Wednesday, April 16, 2014 4:46:10 PM >>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>> >>>>>>> can you try to restart the engine? >>>>>>> This should clean the cache and some of the tables holding temporary >>>>>>> task info. >>>>>>> >>>>>>> Dafna >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 04/16/2014 02:37 PM, Maurice James wrote: >>>>>>>> I ran vdsClient -s 0 getAllTasksInfo and nothing was returned. What >>>>>>>> should >>>>>>>> my next step be? >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>> From: "Dafna Ron" <[email protected]> >>>>>>>> To: "Maurice James" <[email protected]> >>>>>>>> Cc: "Yair Zaslavsky" <[email protected]>, [email protected] >>>>>>>> Sent: Wednesday, April 16, 2014 4:44:48 AM >>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>> >>>>>>>> ok... Now I am starting to understand what's going on and if you look >>>>>>>> at >>>>>>>> the vdsm log, the live snapshot succeeds and no other ERROR are >>>>>>>> reported. I think it's related to the task management on engine side. >>>>>>>> >>>>>>>> Can I ask you to run in the spm: vdsClient -s 0 getAllTasksInfo >>>>>>>> If you have tasks we would have to stop and clear them (vdsClient -s 0 >>>>>>>> stopTask <task> ; vdsClient -s 0 clearTask <task>) >>>>>>>> after you clear the tasks you will have to restart the engine >>>>>>>> >>>>>>>> Also, what version of ovirt are you using? is it 3.4? because this was >>>>>>>> suppose to be fixed... >>>>>>>> >>>>>>>> here is the explanation from what I see in the logs: >>>>>>>> >>>>>>>> you are sending a command to Live Migrate >>>>>>>> >>>>>>>> 2014-04-15 12:50:54,381 INFO >>>>>>>> [org.ovirt.engine.core.bll.MoveDisksCommand] (ajp--127.0.0.1-8702-4) >>>>>>>> [4c089392] Running command: MoveDisksCommand internal: false. Entities >>>>>>>> affected : ID: c24706d3-1872-4cd3-94a2-9c61ef032e29 Type: Disk >>>>>>>> 2014-04-15 12:50:54,520 INFO >>>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>>>>> (ajp--127.0.0.1-8702-4) [4c089392] Lock Acquired to object EngineLock >>>>>>>> [exclusiveLocks= key: c24706d3-1872-4cd3-94a2-9c61ef032e29 value: DISK >>>>>>>> , sharedLocks= key: ba49605b-fb7e-4a70-a380-6286d3903e50 value: VM >>>>>>>> ] >>>>>>>> >>>>>>>> The fist step is creating the snapshot: >>>>>>>> >>>>>>>> 2014-04-15 12:50:54,734 INFO >>>>>>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] >>>>>>>> (org.ovirt.thread.pool-6-thread-47) Running command: >>>>>>>> CreateAllSnapshotsFromVmCommand internal: true. Entities affected : >>>>>>>> ID: >>>>>>>> ba49605b-fb7e-4a70-a3 >>>>>>>> 80-6286d3903e50 Type: VM >>>>>>>> 2014-04-15 12:50:54,748 INFO >>>>>>>> [org.ovirt.engine.core.bll.CreateSnapshotCommand] >>>>>>>> (org.ovirt.thread.pool-6-thread-47) [23c23b4] Running command: >>>>>>>> CreateSnapshotCommand internal: true. Entities affected : ID: >>>>>>>> 00000000-0000-0000-0000-0000000 >>>>>>>> 00000 Type: Storage >>>>>>>> 2014-04-15 12:50:54,760 INFO >>>>>>>> [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] >>>>>>>> (org.ovirt.thread.pool-6-thread-47) [23c23b4] START, >>>>>>>> CreateSnapshotVDSCommand( storagePoolId = >>>>>>>> a106ab81-9d5f-49c1-aeaf-832a137b708c, ignor >>>>>>>> eFailoverLimit = false, storageDomainId = >>>>>>>> 3406665e-4adc-4fd4-aa1e-037547b29adb, imageGroupId = >>>>>>>> c24706d3-1872-4cd3-94a2-9c61ef032e29, imageSizeInBytes = 107374182400, >>>>>>>> volumeFormat = COW, newImageId = 015f7d9d-ff75-4a3c-a634-a00e82e04803, >>>>>>>> newImageDescription = , imageId = e8442348-e28e-4e78-abe8-4f2848b47661, >>>>>>>> sourceImageGroupId = c24706d3-1872-4cd3-94a2-9c61ef032e29), log id: >>>>>>>> f3b7d8b >>>>>>>> >>>>>>>> >>>>>>>> which actually succeeds in vdsm and reported as successful to the >>>>>>>> engine: >>>>>>>> >>>>>>>> engine: >>>>>>>> >>>>>>>> 2014-04-15 12:51:07,291 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] >>>>>>>> (DefaultQuartzScheduler_Worker-42) SPMAsyncTask::PollTask: Polling task >>>>>>>> db7f0f4d-f47d-472a-92bd-197b428fe417 (Parent Command >>>>>>>> LiveMigrateVmDisks, >>>>>>>> Parameters Type >>>>>>>> org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned >>>>>>>> status finished, result 'success'. >>>>>>>> >>>>>>>> vdsm: >>>>>>>> >>>>>>>> Thread-4670::DEBUG::2014-04-15 >>>>>>>> 12:51:07,287::task::1185::TaskManager.Task::(prepare) >>>>>>>> Task=`ea9d5846-656c-4a44-bca2-d5c091004078`::finished: >>>>>>>> {'allTasksStatus': {'db7f0f4d-f47d-472a-92bd-197b428fe417': {'code': 0, >>>>>>>> 'message': '1 jobs comple >>>>>>>> ted successfully', 'taskState': 'finished', 'taskResult': 'success', >>>>>>>> 'taskID': 'db7f0f4d-f47d-472a-92bd-197b428fe417'}}} >>>>>>>> >>>>>>>> >>>>>>>> now engine has to end the task: >>>>>>>> >>>>>>>> 2014-04-15 12:51:07,301 INFO >>>>>>>> [org.ovirt.engine.core.bll.CommandAsyncTask] >>>>>>>> (DefaultQuartzScheduler_Worker-42) CommandAsyncTask::endAction: Ending >>>>>>>> action for 1 tasks (command ID: 7c67ca08-9969-4a6e-87ef-0b7379d7edb6): >>>>>>>> calling endAction . >>>>>>>> 2014-04-15 12:51:07,303 INFO >>>>>>>> [org.ovirt.engine.core.bll.CommandAsyncTask] >>>>>>>> (org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::EndCommandAction >>>>>>>> [within thread] context: Attempting to endAction LiveMigrateVmDisks, >>>>>>>> executionIndex: 0 >>>>>>>> 2014-04-15 12:51:07,315 INFO >>>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>>>>> (org.ovirt.thread.pool-6-thread-42) Ending command successfully: >>>>>>>> org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand >>>>>>>> 2014-04-15 12:51:07,319 INFO >>>>>>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] >>>>>>>> (org.ovirt.thread.pool-6-thread-42) Ending command successfully: >>>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand >>>>>>>> >>>>>>>> and than >>>>>>>> >>>>>>>> we see an exception: >>>>>>>> >>>>>>>> 2014-04-15 12:51:07,498 WARN >>>>>>>> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] >>>>>>>> (org.ovirt.thread.pool-6-thread-42) Wasnt able to live snapshot due to >>>>>>>> error: VdcBLLException: VdcBLLException: >>>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: >>>>>>>> VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = >>>>>>>> Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code >>>>>>>> 48). VM will still be configured to the new created snapshot >>>>>>>> 2014-04-15 12:51:07,543 INFO >>>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>>> (org.ovirt.thread.pool-6-thread-42) Correlation ID: null, Call Stack: >>>>>>>> org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: >>>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: >>>>>>>> VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error = >>>>>>>> Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code >>>>>>>> 48) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1971) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransaction(CreateAllSnapshotsFromVmCommand.java:354) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransaction(CreateAllSnapshotsFromVmCommand.java:351) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.performLiveSnapshot(CreateAllSnapshotsFromVmCommand.java:351) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.endVmCommand(CreateAllSnapshotsFromVmCommand.java:273) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.VmCommand.endSuccessfully(VmCommand.java:304) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.CommandBase.internalEndSuccessfully(CommandBase.java:614) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:560) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1886) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:151) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:118) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:492) >>>>>>>> at >>>>>>>> org.ovirt.engine.core.bll.Backend.endAction(Backend.java:446) >>>>>>>> at >>>>>>>> sun.reflect.GeneratedMethodAccessor513.invoke(Unknown Source) >>>>>>>> at >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 04/15/2014 05:54 PM, Maurice James wrote: >>>>>>>>> Logs are attached. >>>>>>>>> >>>>>>>>> Live Migration failed >>>>>>>>> >>>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Dafna Ron" <[email protected]> >>>>>>>>> To: "Maurice James" <[email protected]> >>>>>>>>> Cc: "Yair Zaslavsky" <[email protected]>, [email protected] >>>>>>>>> Sent: Tuesday, April 15, 2014 10:46:29 AM >>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>> >>>>>>>>> yeah, but is it the same error we see before in vdsm? :) >>>>>>>>> lets try to follow the action in both logs so we can debug it better, >>>>>>>>> can you do the following? >>>>>>>>> >>>>>>>>> 1. stop the vm and clean all snapshots. >>>>>>>>> 2. add >>>> to the engine and vdsm logs using echo (#echo '>>>>>>>>' >>>>>>>>> >> >>>>>>>>> /var/log/ovirt-engine/engine.log and #echo '>>>>>>>>' >> >>>>>>>>> /var/log/vdsm/vdsm.log) >>>>>>>>> 3. start the vm >>>>>>>>> 4. try to live migrate the disk >>>>>>>>> 5. attach the full engine and vdsm logs. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Dafna >>>>>>>>> >>>>>>>>> >>>>>>>>> On 04/15/2014 03:41 PM, Maurice James wrote: >>>>>>>>>> It failed again. Its reported in the same location. >>>>>>>>>> >>>>>>>>>> Snip of engine.log >>>>>>>>>> >>>>>>>>>> 2014-04-15 10:41:17,174 ERROR >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] >>>>>>>>>> (org.ovirt.thread.pool-6-thread-34) Failed in SnapshotVDS method >>>>>>>>>> 2014-04-15 10:41:17,178 ERROR >>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] >>>>>>>>>> (org.ovirt.thread.pool-6-thread-34) Command >>>>>>>>>> SnapshotVDSCommand(HostName >>>>>>>>>> = vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195, >>>>>>>>>> vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. >>>>>>>>>> Exception: >>>>>>>>>> VDSErrorException: VDSGenericException: VDSErrorException: Failed to >>>>>>>>>> SnapshotVDS, error = Snapshot failed, code = 48 >>>>>>>>>> 2014-04-15 10:41:17,531 ERROR >>>>>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>>>>>>> (org.ovirt.thread.pool-6-thread-34) Command >>>>>>>>>> org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll >>>>>>>>>> exception. With error message VdcBLLException: Auto-generated live >>>>>>>>>> snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed >>>>>>>>>> with >>>>>>>>>> error imageErr and code 13) >>>>>>>>>> 2014-04-15 10:41:17,534 ERROR >>>>>>>>>> [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] >>>>>>>>>> (org.ovirt.thread.pool-6-thread-34) Reverting task unknown, handler: >>>>>>>>>> org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Dafna Ron" <[email protected]> >>>>>>>>>> To: "Maurice James" <[email protected]> >>>>>>>>>> Cc: "Yair Zaslavsky" <[email protected]>, [email protected] >>>>>>>>>> Sent: Tuesday, April 15, 2014 10:37:41 AM >>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>> >>>>>>>>>> ok. >>>>>>>>>> so the vm was down and you deleted all the snapshots. >>>>>>>>>> in the webadmin, when you select the disk you want to live migrate, >>>>>>>>>> is >>>>>>>>>> it the same storage reported in vdsm? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Dafna >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 04/15/2014 03:25 PM, Maurice James wrote: >>>>>>>>>>> I restarted from the VM. The VM now has 0 snapshots. All of the >>>>>>>>>>> snapshots in the screen shots never completed, so I deleted them >>>>>>>>>>> while >>>>>>>>>>> the VM was powerd off from the ui >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Dafna Ron" <[email protected]> >>>>>>>>>>> To: "Maurice James" <[email protected]> >>>>>>>>>>> Cc: "Yair Zaslavsky" <[email protected]>, [email protected] >>>>>>>>>>> Sent: Tuesday, April 15, 2014 10:21:06 AM >>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>> >>>>>>>>>>> did you restart the vm from the vm internally or stop -> start the >>>>>>>>>>> vm? >>>>>>>>>>> >>>>>>>>>>> how many snapshots does the vm have? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Dafna >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 04/15/2014 03:16 PM, Maurice James wrote: >>>>>>>>>>>> I restarted the VM. Same issue. I was able to find the physical >>>>>>>>>>>> location on disk. I attached 2 screen shots showing the location of >>>>>>>>>>>> the disks on the file system. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Dafna Ron" <[email protected]> >>>>>>>>>>>> To: "Maurice James" <[email protected]> >>>>>>>>>>>> Cc: "Yair Zaslavsky" <[email protected]>, [email protected] >>>>>>>>>>>> Sent: Tuesday, April 15, 2014 10:12:56 AM >>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>> >>>>>>>>>>>> it seems that the issue is the links between the engine and the >>>>>>>>>>>> vdsm >>>>>>>>>>>> >>>>>>>>>>>> why don't you check where the disk is physically located and than >>>>>>>>>>>> try >>>>>>>>>>>> to >>>>>>>>>>>> restart the vm? >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Dafna >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 04/15/2014 02:57 PM, Maurice James wrote: >>>>>>>>>>>>> yes I did see a warning about restarting. It said: >>>>>>>>>>>>> Failed to create live snapshot '20140415 for VM >>>>>>>>>>>>> 'TIEATS_Racktables'. >>>>>>>>>>>>> VM restart is recommended >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Dafna Ron" <[email protected]> >>>>>>>>>>>>> To: "Maurice James" <[email protected]> >>>>>>>>>>>>> Cc: "Yair Zaslavsky" <[email protected]>, [email protected] >>>>>>>>>>>>> Sent: Tuesday, April 15, 2014 9:49:22 AM >>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Maurice, >>>>>>>>>>>>> >>>>>>>>>>>>> are you getting any errors/alert about restarting the vm in the >>>>>>>>>>>>> event >>>>>>>>>>>>> log (in the webadmin). >>>>>>>>>>>>> >>>>>>>>>>>>> I believe that you have a mismatch in the directory link between >>>>>>>>>>>>> the >>>>>>>>>>>>> db >>>>>>>>>>>>> and vdsm which was created when you tried to live migrate and >>>>>>>>>>>>> failed >>>>>>>>>>>>> the >>>>>>>>>>>>> fist time). >>>>>>>>>>>>> >>>>>>>>>>>>> we can see in the vdsm log that createVolume (which is part of the >>>>>>>>>>>>> live >>>>>>>>>>>>> migration) is failing with this error: >>>>>>>>>>>>> >>>>>>>>>>>>> OSError: [Errno 2] No such file or directory: >>>>>>>>>>>>> '/rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/b7663d70-e658-41fa-b9f0-8da83c9eddce/images/348effd9-c5db-44fc-a9e5-67391647096b' >>>>>>>>>>>>> >>>>>>>>>>>>> so it seems that the engine is pointing to a link that does not >>>>>>>>>>>>> exist >>>>>>>>>>>>> and we need to find on what storage the disk actually exists on >>>>>>>>>>>>> and >>>>>>>>>>>>> where the engine is pointing to. >>>>>>>>>>>>> >>>>>>>>>>>>> you can look at the webadmin at the disk and see where it's >>>>>>>>>>>>> suppose to >>>>>>>>>>>>> be and than look in /rhev and see where the disk link is in vdsm >>>>>>>>>>>>> (if >>>>>>>>>>>>> you >>>>>>>>>>>>> can see where the disk is actually is - physically in the storage, >>>>>>>>>>>>> that >>>>>>>>>>>>> would be good too). >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Dafna >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 04/15/2014 01:16 PM, Maurice James wrote: >>>>>>>>>>>>>> As far as the disk creation , I got that sorted out. It seem >>>>>>>>>>>>>> that my >>>>>>>>>>>>>> vdsm versions were out of sync on my hosts. When I updated them >>>>>>>>>>>>>> the >>>>>>>>>>>>>> disk creation began working, but im still having issues with live >>>>>>>>>>>>>> disk migrations. >>>>>>>>>>>>>> >>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>> From: "Dafna Ron" <[email protected]> >>>>>>>>>>>>>> To: "Maurice James" <[email protected]> >>>>>>>>>>>>>> Cc: "Yair Zaslavsky" <[email protected]>, [email protected] >>>>>>>>>>>>>> Sent: Tuesday, April 15, 2014 5:31:48 AM >>>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also, before the task fails, there are ERRORs in the log which >>>>>>>>>>>>>> are >>>>>>>>>>>>>> reporting problems in the DC >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2014-04-14 09:56:05,161 ERROR >>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-16) Command >>>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>>>>> connect timed out >>>>>>>>>>>>>> 2014-04-14 09:56:10,226 ERROR >>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-49) Command >>>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>>>>> connect timed out >>>>>>>>>>>>>> 2014-04-14 09:56:15,283 ERROR >>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-81) Command >>>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>>>>> connect timed out >>>>>>>>>>>>>> 2014-04-14 09:56:20,342 ERROR >>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-58) Command >>>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>>>>> connect timed out >>>>>>>>>>>>>> 2014-04-14 09:56:25,409 ERROR >>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-84) Command >>>>>>>>>>>>>> GetCapabilitiesVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution >>>>>>>>>>>>>> failed. Exception: VDSNetworkException: >>>>>>>>>>>>>> java.net.SocketTimeoutException: >>>>>>>>>>>>>> connect timed out >>>>>>>>>>>>>> 2014-04-14 09:56:30,542 INFO >>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] >>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-91) START, >>>>>>>>>>>>>> GetHardwareInfoVDSCommand(HostName = vhost3, HostId = >>>>>>>>>>>>>> bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]), log id: >>>>>>>>>>>>>> 30f312c >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> than you stop the vm and fail the AddDisk. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> is there a chance you were having problems with your storage? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dafna >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 04/14/2014 04:04 PM, Maurice James wrote: >>>>>>>>>>>>>>> Snapshot creation seems to be failing as well >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>> From: "Yair Zaslavsky" <[email protected]> >>>>>>>>>>>>>>> To: "Maurice James" <[email protected]>, "Federico >>>>>>>>>>>>>>> Simoncelli" >>>>>>>>>>>>>>> <[email protected]> >>>>>>>>>>>>>>> Cc: [email protected] >>>>>>>>>>>>>>> Sent: Monday, April 14, 2014 11:00:55 AM >>>>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Federico, >>>>>>>>>>>>>>> Can you please take a look? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>> From: "Maurice James" <[email protected]> >>>>>>>>>>>>>>>> To: "Yair Zaslavsky" <[email protected]> >>>>>>>>>>>>>>>> Cc: [email protected] >>>>>>>>>>>>>>>> Sent: Monday, April 14, 2014 5:44:44 PM >>>>>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Logs attached >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>> From: "Yair Zaslavsky" <[email protected]> >>>>>>>>>>>>>>>> To: "Maurice James" <[email protected]> >>>>>>>>>>>>>>>> Cc: [email protected] >>>>>>>>>>>>>>>> Sent: Monday, April 14, 2014 10:33:03 AM >>>>>>>>>>>>>>>> Subject: Re: [ovirt-users] Error creating Disks >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> IMHO not enough info is provided, >>>>>>>>>>>>>>>> Can you please provide full engine.log and relevant vdsm.log? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> THanks, >>>>>>>>>>>>>>>> Yair >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>>> From: "Maurice James" <[email protected]> >>>>>>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>>>>>> Sent: Monday, April 14, 2014 5:00:37 PM >>>>>>>>>>>>>>>>> Subject: [ovirt-users] Error creating Disks >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> oVirt Engine Version: >>>>>>>>>>>>>>>>> 3.4.1-0.0.master.20140412010845.git43746c6.el6 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> While attempting to create a disk on an NFS storage domain, it >>>>>>>>>>>>>>>>> fails with >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> following error in the engine.log >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2014-04-14 09:58:12,127 ERROR >>>>>>>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] >>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-72) Failed in >>>>>>>>>>>>>>>>> HSMGetAllTasksStatusesVDS >>>>>>>>>>>>>>>>> method >>>>>>>>>>>>>>>>> 2014-04-14 09:58:12,139 ERROR >>>>>>>>>>>>>>>>> [org.ovirt.engine.core.bll.SPMAsyncTask] >>>>>>>>>>>>>>>>> (DefaultQuartzScheduler_Worker-72) >>>>>>>>>>>>>>>>> BaseAsyncTask::LogEndTaskFailure: Task >>>>>>>>>>>>>>>>> ee6ce682-bd76-467a-82d2-d227229cb9de (Parent Command AddDisk, >>>>>>>>>>>>>>>>> Parameters >>>>>>>>>>>>>>>>> Type >>>>>>>>>>>>>>>>> org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) >>>>>>>>>>>>>>>>> ended >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>> failure: >>>>>>>>>>>>>>>>> 2014-04-14 09:58:12,159 ERROR >>>>>>>>>>>>>>>>> [org.ovirt.engine.core.bll.AddDiskCommand] >>>>>>>>>>>>>>>>> (org.ovirt.thread.pool-6-thread-9) [483e53d6] Ending command >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>> failure: >>>>>>>>>>>>>>>>> org.ovirt.engine.core.bll.AddDiskCommand >>>>>>>>>>>>>>>>> 2014-04-14 09:58:12,212 ERROR >>>>>>>>>>>>>>>>> [org.ovirt.engine.core.bll.AddImageFromScratchCommand] >>>>>>>>>>>>>>>>> (org.ovirt.thread.pool-6-thread-9) [ab1e0be] Ending command >>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>> failure: >>>>>>>>>>>>>>>>> org.ovirt.engine.core.bll.AddImageFromScratchCommand >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Users mailing list >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> -- >>>>>>> Dafna Ron >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> [email protected] >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> > -- Dafna Ron _______________________________________________ Users mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/users
