I ran vdsClient -s 0 getAllTasksInfo and nothing was returned. What should
my next step be?
----- Original Message -----
From: "Dafna Ron" <d...@redhat.com>
To: "Maurice James" <mja...@media-node.com>
Cc: "Yair Zaslavsky" <yzasl...@redhat.com>, users@ovirt.org
Sent: Wednesday, April 16, 2014 4:44:48 AM
Subject: Re: [ovirt-users] Error creating Disks
ok... Now I am starting to understand what's going on and if you look at
the vdsm log, the live snapshot succeeds and no other ERROR are
reported. I think it's related to the task management on engine side.
Can I ask you to run in the spm: vdsClient -s 0 getAllTasksInfo
If you have tasks we would have to stop and clear them (vdsClient -s 0
stopTask <task> ; vdsClient -s 0 clearTask <task>)
after you clear the tasks you will have to restart the engine
Also, what version of ovirt are you using? is it 3.4? because this was
suppose to be fixed...
here is the explanation from what I see in the logs:
you are sending a command to Live Migrate
2014-04-15 12:50:54,381 INFO
[org.ovirt.engine.core.bll.MoveDisksCommand] (ajp--127.0.0.1-8702-4)
[4c089392] Running command: MoveDisksCommand internal: false. Entities
affected : ID: c24706d3-1872-4cd3-94a2-9c61ef032e29 Type: Disk
2014-04-15 12:50:54,520 INFO
[org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand]
(ajp--127.0.0.1-8702-4) [4c089392] Lock Acquired to object EngineLock
[exclusiveLocks= key: c24706d3-1872-4cd3-94a2-9c61ef032e29 value: DISK
, sharedLocks= key: ba49605b-fb7e-4a70-a380-6286d3903e50 value: VM
]
The fist step is creating the snapshot:
2014-04-15 12:50:54,734 INFO
[org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand]
(org.ovirt.thread.pool-6-thread-47) Running command:
CreateAllSnapshotsFromVmCommand internal: true. Entities affected : ID:
ba49605b-fb7e-4a70-a3
80-6286d3903e50 Type: VM
2014-04-15 12:50:54,748 INFO
[org.ovirt.engine.core.bll.CreateSnapshotCommand]
(org.ovirt.thread.pool-6-thread-47) [23c23b4] Running command:
CreateSnapshotCommand internal: true. Entities affected : ID:
00000000-0000-0000-0000-0000000
00000 Type: Storage
2014-04-15 12:50:54,760 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand]
(org.ovirt.thread.pool-6-thread-47) [23c23b4] START,
CreateSnapshotVDSCommand( storagePoolId =
a106ab81-9d5f-49c1-aeaf-832a137b708c, ignor
eFailoverLimit = false, storageDomainId =
3406665e-4adc-4fd4-aa1e-037547b29adb, imageGroupId =
c24706d3-1872-4cd3-94a2-9c61ef032e29, imageSizeInBytes = 107374182400,
volumeFormat = COW, newImageId = 015f7d9d-ff75-4a3c-a634-a00e82e04803,
newImageDescription = , imageId = e8442348-e28e-4e78-abe8-4f2848b47661,
sourceImageGroupId = c24706d3-1872-4cd3-94a2-9c61ef032e29), log id: f3b7d8b
which actually succeeds in vdsm and reported as successful to the engine:
engine:
2014-04-15 12:51:07,291 INFO [org.ovirt.engine.core.bll.SPMAsyncTask]
(DefaultQuartzScheduler_Worker-42) SPMAsyncTask::PollTask: Polling task
db7f0f4d-f47d-472a-92bd-197b428fe417 (Parent Command LiveMigrateVmDisks,
Parameters Type
org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned
status finished, result 'success'.
vdsm:
Thread-4670::DEBUG::2014-04-15
12:51:07,287::task::1185::TaskManager.Task::(prepare)
Task=`ea9d5846-656c-4a44-bca2-d5c091004078`::finished:
{'allTasksStatus': {'db7f0f4d-f47d-472a-92bd-197b428fe417': {'code': 0,
'message': '1 jobs comple
ted successfully', 'taskState': 'finished', 'taskResult': 'success',
'taskID': 'db7f0f4d-f47d-472a-92bd-197b428fe417'}}}
now engine has to end the task:
2014-04-15 12:51:07,301 INFO
[org.ovirt.engine.core.bll.CommandAsyncTask]
(DefaultQuartzScheduler_Worker-42) CommandAsyncTask::endAction: Ending
action for 1 tasks (command ID: 7c67ca08-9969-4a6e-87ef-0b7379d7edb6):
calling endAction .
2014-04-15 12:51:07,303 INFO
[org.ovirt.engine.core.bll.CommandAsyncTask]
(org.ovirt.thread.pool-6-thread-42) CommandAsyncTask::EndCommandAction
[within thread] context: Attempting to endAction LiveMigrateVmDisks,
executionIndex: 0
2014-04-15 12:51:07,315 INFO
[org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand]
(org.ovirt.thread.pool-6-thread-42) Ending command successfully:
org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand
2014-04-15 12:51:07,319 INFO
[org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand]
(org.ovirt.thread.pool-6-thread-42) Ending command successfully:
org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand
and than
we see an exception:
2014-04-15 12:51:07,498 WARN
[org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand]
(org.ovirt.thread.pool-6-thread-42) Wasnt able to live snapshot due to
error: VdcBLLException: VdcBLLException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error =
Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code
48). VM will still be configured to the new created snapshot
2014-04-15 12:51:07,543 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-42) Correlation ID: null, Call Stack:
org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to SnapshotVDS, error =
Snapshot failed, code = 48 (Failed with error SNAPSHOT_FAILED and code 48)
at
org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:116)
at
org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33)
at
org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1971)
at
org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransaction(CreateAllSnapshotsFromVmCommand.java:354)
at
org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand$2.runInTransaction(CreateAllSnapshotsFromVmCommand.java:351)
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174)
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116)
at
org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.performLiveSnapshot(CreateAllSnapshotsFromVmCommand.java:351)
at
org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand.endVmCommand(CreateAllSnapshotsFromVmCommand.java:273)
at
org.ovirt.engine.core.bll.VmCommand.endSuccessfully(VmCommand.java:304)
at
org.ovirt.engine.core.bll.CommandBase.internalEndSuccessfully(CommandBase.java:614)
at
org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:560)
at
org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1886)
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:151)
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:118)
at
org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:492)
at org.ovirt.engine.core.bll.Backend.endAction(Backend.java:446)
at sun.reflect.GeneratedMethodAccessor513.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
On 04/15/2014 05:54 PM, Maurice James wrote:
Logs are attached.
Live Migration failed
----- Original Message -----
From: "Dafna Ron" <d...@redhat.com>
To: "Maurice James" <mja...@media-node.com>
Cc: "Yair Zaslavsky" <yzasl...@redhat.com>, users@ovirt.org
Sent: Tuesday, April 15, 2014 10:46:29 AM
Subject: Re: [ovirt-users] Error creating Disks
yeah, but is it the same error we see before in vdsm? :)
lets try to follow the action in both logs so we can debug it better,
can you do the following?
1. stop the vm and clean all snapshots.
2. add >>>> to the engine and vdsm logs using echo (#echo '>>>>>>>>' >>
/var/log/ovirt-engine/engine.log and #echo '>>>>>>>>' >>
/var/log/vdsm/vdsm.log)
3. start the vm
4. try to live migrate the disk
5. attach the full engine and vdsm logs.
Thanks,
Dafna
On 04/15/2014 03:41 PM, Maurice James wrote:
It failed again. Its reported in the same location.
Snip of engine.log
2014-04-15 10:41:17,174 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
(org.ovirt.thread.pool-6-thread-34) Failed in SnapshotVDS method
2014-04-15 10:41:17,178 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
(org.ovirt.thread.pool-6-thread-34) Command SnapshotVDSCommand(HostName
= vhost3, HostId = bc9c25e6-714e-4eac-8af0-860ac76fd195,
vmId=ba49605b-fb7e-4a70-a380-6286d3903e50) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SnapshotVDS, error = Snapshot failed, code = 48
2014-04-15 10:41:17,531 ERROR
[org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand]
(org.ovirt.thread.pool-6-thread-34) Command
org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand throw Vdc Bll
exception. With error message VdcBLLException: Auto-generated live
snapshot for VM ba49605b-fb7e-4a70-a380-6286d3903e50 failed (Failed with
error imageErr and code 13)
2014-04-15 10:41:17,534 ERROR
[org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand]
(org.ovirt.thread.pool-6-thread-34) Reverting task unknown, handler:
org.ovirt.engine.core.bll.lsm.LiveSnapshotTaskHandler
----- Original Message -----
From: "Dafna Ron" <d...@redhat.com>
To: "Maurice James" <mja...@media-node.com>
Cc: "Yair Zaslavsky" <yzasl...@redhat.com>, users@ovirt.org
Sent: Tuesday, April 15, 2014 10:37:41 AM
Subject: Re: [ovirt-users] Error creating Disks
ok.
so the vm was down and you deleted all the snapshots.
in the webadmin, when you select the disk you want to live migrate, is
it the same storage reported in vdsm?
Thanks,
Dafna
On 04/15/2014 03:25 PM, Maurice James wrote:
I restarted from the VM. The VM now has 0 snapshots. All of the
snapshots in the screen shots never completed, so I deleted them while
the VM was powerd off from the ui
----- Original Message -----
From: "Dafna Ron" <d...@redhat.com>
To: "Maurice James" <mja...@media-node.com>
Cc: "Yair Zaslavsky" <yzasl...@redhat.com>, users@ovirt.org
Sent: Tuesday, April 15, 2014 10:21:06 AM
Subject: Re: [ovirt-users] Error creating Disks
did you restart the vm from the vm internally or stop -> start the vm?
how many snapshots does the vm have?
Thanks,
Dafna
On 04/15/2014 03:16 PM, Maurice James wrote:
I restarted the VM. Same issue. I was able to find the physical
location on disk. I attached 2 screen shots showing the location of
the disks on the file system.
----- Original Message -----
From: "Dafna Ron" <d...@redhat.com>
To: "Maurice James" <mja...@media-node.com>
Cc: "Yair Zaslavsky" <yzasl...@redhat.com>, users@ovirt.org
Sent: Tuesday, April 15, 2014 10:12:56 AM
Subject: Re: [ovirt-users] Error creating Disks
it seems that the issue is the links between the engine and the vdsm
why don't you check where the disk is physically located and than try
to
restart the vm?
Thanks,
Dafna
On 04/15/2014 02:57 PM, Maurice James wrote:
yes I did see a warning about restarting. It said:
Failed to create live snapshot '20140415 for VM 'TIEATS_Racktables'.
VM restart is recommended
----- Original Message -----
From: "Dafna Ron" <d...@redhat.com>
To: "Maurice James" <mja...@media-node.com>
Cc: "Yair Zaslavsky" <yzasl...@redhat.com>, users@ovirt.org
Sent: Tuesday, April 15, 2014 9:49:22 AM
Subject: Re: [ovirt-users] Error creating Disks
Hi Maurice,
are you getting any errors/alert about restarting the vm in the event
log (in the webadmin).
I believe that you have a mismatch in the directory link between the
db
and vdsm which was created when you tried to live migrate and failed
the
fist time).
we can see in the vdsm log that createVolume (which is part of the
live
migration) is failing with this error:
OSError: [Errno 2] No such file or directory:
'/rhev/data-center/a106ab81-9d5f-49c1-aeaf-832a137b708c/b7663d70-e658-41fa-b9f0-8da83c9eddce/images/348effd9-c5db-44fc-a9e5-67391647096b'
so it seems that the engine is pointing to a link that does not exist
and we need to find on what storage the disk actually exists on and
where the engine is pointing to.
you can look at the webadmin at the disk and see where it's suppose to
be and than look in /rhev and see where the disk link is in vdsm (if
you
can see where the disk is actually is - physically in the storage,
that
would be good too).
Thanks,
Dafna
On 04/15/2014 01:16 PM, Maurice James wrote:
As far as the disk creation , I got that sorted out. It seem that my
vdsm versions were out of sync on my hosts. When I updated them the
disk creation began working, but im still having issues with live
disk migrations.
----- Original Message -----
From: "Dafna Ron" <d...@redhat.com>
To: "Maurice James" <mja...@media-node.com>
Cc: "Yair Zaslavsky" <yzasl...@redhat.com>, users@ovirt.org
Sent: Tuesday, April 15, 2014 5:31:48 AM
Subject: Re: [ovirt-users] Error creating Disks
Also, before the task fails, there are ERRORs in the log which are
reporting problems in the DC
2014-04-14 09:56:05,161 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-16) Command
GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
failed. Exception: VDSNetworkException:
java.net.SocketTimeoutException:
connect timed out
2014-04-14 09:56:10,226 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-49) Command
GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
failed. Exception: VDSNetworkException:
java.net.SocketTimeoutException:
connect timed out
2014-04-14 09:56:15,283 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-81) Command
GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
failed. Exception: VDSNetworkException:
java.net.SocketTimeoutException:
connect timed out
2014-04-14 09:56:20,342 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-58) Command
GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
failed. Exception: VDSNetworkException:
java.net.SocketTimeoutException:
connect timed out
2014-04-14 09:56:25,409 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-84) Command
GetCapabilitiesVDSCommand(HostName = vhost3, HostId =
bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]) execution
failed. Exception: VDSNetworkException:
java.net.SocketTimeoutException:
connect timed out
2014-04-14 09:56:30,542 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
(DefaultQuartzScheduler_Worker-91) START,
GetHardwareInfoVDSCommand(HostName = vhost3, HostId =
bc9c25e6-714e-4eac-8af0-860ac76fd195, vds=Host[vhost3]), log id:
30f312c
than you stop the vm and fail the AddDisk.
is there a chance you were having problems with your storage?
Dafna
On 04/14/2014 04:04 PM, Maurice James wrote:
Snapshot creation seems to be failing as well
----- Original Message -----
From: "Yair Zaslavsky" <yzasl...@redhat.com>
To: "Maurice James" <mja...@media-node.com>, "Federico Simoncelli"
<fsimo...@redhat.com>
Cc: users@ovirt.org
Sent: Monday, April 14, 2014 11:00:55 AM
Subject: Re: [ovirt-users] Error creating Disks
Hi Federico,
Can you please take a look?
----- Original Message -----
From: "Maurice James" <mja...@media-node.com>
To: "Yair Zaslavsky" <yzasl...@redhat.com>
Cc: users@ovirt.org
Sent: Monday, April 14, 2014 5:44:44 PM
Subject: Re: [ovirt-users] Error creating Disks
Logs attached
----- Original Message -----
From: "Yair Zaslavsky" <yzasl...@redhat.com>
To: "Maurice James" <mja...@media-node.com>
Cc: users@ovirt.org
Sent: Monday, April 14, 2014 10:33:03 AM
Subject: Re: [ovirt-users] Error creating Disks
Hi,
IMHO not enough info is provided,
Can you please provide full engine.log and relevant vdsm.log?
THanks,
Yair
----- Original Message -----
From: "Maurice James" <mja...@media-node.com>
To: users@ovirt.org
Sent: Monday, April 14, 2014 5:00:37 PM
Subject: [ovirt-users] Error creating Disks
oVirt Engine Version:
3.4.1-0.0.master.20140412010845.git43746c6.el6
While attempting to create a disk on an NFS storage domain, it
fails with
the
following error in the engine.log
2014-04-14 09:58:12,127 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-72) Failed in
HSMGetAllTasksStatusesVDS
method
2014-04-14 09:58:12,139 ERROR
[org.ovirt.engine.core.bll.SPMAsyncTask]
(DefaultQuartzScheduler_Worker-72)
BaseAsyncTask::LogEndTaskFailure: Task
ee6ce682-bd76-467a-82d2-d227229cb9de (Parent Command AddDisk,
Parameters
Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters)
ended
with
failure:
2014-04-14 09:58:12,159 ERROR
[org.ovirt.engine.core.bll.AddDiskCommand]
(org.ovirt.thread.pool-6-thread-9) [483e53d6] Ending command with
failure:
org.ovirt.engine.core.bll.AddDiskCommand
2014-04-14 09:58:12,212 ERROR
[org.ovirt.engine.core.bll.AddImageFromScratchCommand]
(org.ovirt.thread.pool-6-thread-9) [ab1e0be] Ending command with
failure:
org.ovirt.engine.core.bll.AddImageFromScratchCommand
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
