On 11/06/2015 02:02 PM, Simone Tiraboschi wrote:
On Fri, Nov 6, 2015 at 5:21 PM, Jason Keltz <j...@cse.yorku.ca
<mailto:j...@cse.yorku.ca>> wrote:
Hi.
Last night, I upgraded my engine from 3.5 to 3.6. That went
flawlessly.
Today, I'm trying to upgrade the vdsm on the hosts from 3.5 to 3.6
(along with applying other RHEL7.1 updates) However, when I'm
trying to put each host into maintenance mode, and migrations
start to occur, they all seem to FAIL now! Even worse, when they
fail, it leaves the hosts DOWN! If there's a failure, I'd expect
the host to simply abort the migration.... Any help in debugging
this would be VERY much appreciated!
2015-11-06 10:09:16,065 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-4) [] Correlation ID:
658ba478, Job ID: 524e8c44-04e0-42d3-89f9-9f4e4d397583, Call
Stack: null, Custom Event ID: -1, Message: Migration failed
(VM: eportfolio, Source: virt1).
2015-11-06 10:10:17,112 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-22) [2f0dee16] Correlation ID:
7da3ac1b, Job ID: 93c0b1f2-4c8e-48cf-9e63-c1ba91be425f, Call
Stack: null, Custom Event ID: -1, Message: Migration failed
(VM: ftp1, Source: virt1).
2015-11-06 10:15:08,273 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-45) [] Correlation ID:
5394ef76, Job ID: 994065fc-a142-4821-934a-c2297d86ec12, Call
Stack: null, Custom Event ID: -1, Message: Migration failed
while Host is in 'preparing for maintenance' state.
2015-11-06 10:19:13,712 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-36) [] Correlation ID:
6e422728, Job ID: 994065fc-a142-4821-934a-c2297d86ec12, Call
Stack: null, Custom Event ID: -1, Message: Migration failed
while Host is in 'preparing for maintenance' state.
2015-11-06 10:42:37,852 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-12) [] Correlation ID:
e7f6300, Job ID: 1ea16622-0fa0-4e92-89e5-9dc235c03ef8, Call
Stack: null, Custom Event ID: -1, Message: Migration failed
(VM: ipa, Source: virt1).
2015-11-06 10:43:59,732 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-40) [] Correlation ID:
39cfdf9, Job ID: 72be29bc-a02b-4a90-b5ec-8b995c2fa692, Call
Stack: null, Custom Event ID: -1, Message: Migration failed
(VM: labtesteval, Source: virt1).
2015-11-06 10:52:11,893 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-8-thread-23) [] Correlation ID:
5c435149, Job ID: 1dcd1e14-baa6-44bc-a853-5d33107b759c, Call
Stack: null, Custom Event ID: -1, Message: Migration failed
(VM: www-vhost, Source: virt1).
The complete engine log, virt1, virt2, and virt3 vdsm logs are here:
http://www.eecs.yorku.ca/~jas/ovirt-debug/11062015
<http://www.eecs.yorku.ca/%7Ejas/ovirt-debug/11062015>
Is vdsmd service still active on that hosts?
Hi Simone..
Yes..
virt1:
sh-4.2# systemctl -l status vdsmd
vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
Active: active (running) since Thu 2015-11-05 22:47:46 EST; 15h ago
Main PID: 16520 (vdsm)
CGroup: /system.slice/vdsmd.service
├─16520 /usr/bin/python /usr/share/vdsm/vdsm
├─30038 /usr/libexec/ioprocess --read-pipe-fd 67
--write-pipe-fd 66 --max-threads 10 --max-queued-requests 10
├─30055 /usr/libexec/ioprocess --read-pipe-fd 76
--write-pipe-fd 75 --max-threads 10 --max-queued-requests 10
└─30062 /usr/libexec/ioprocess --read-pipe-fd 81
--write-pipe-fd 84 --max-threads 10 --max-queued-requests 10
Nov 06 10:09:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.com.redhat.rhevm.vdsm
already removed
Nov 06 10:09:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.org.qemu.guest_agent.0
already removed
Nov 06 10:10:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.com.redhat.rhevm.vdsm
already removed
Nov 06 10:10:15 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.org.qemu.guest_agent.0
already removed
Nov 06 10:42:36 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.com.redhat.rhevm.vdsm
already removed
Nov 06 10:42:36 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.org.qemu.guest_agent.0
already removed
Nov 06 10:43:57 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.com.redhat.rhevm.vdsm
already removed
Nov 06 10:43:57 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.org.qemu.guest_agent.0
already removed
Nov 06 10:52:08 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.com.redhat.rhevm.vdsm
already removed
Nov 06 10:52:08 virt1.cs.yorku.ca vdsm[16520]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.org.qemu.guest_agent.0
already removed
virt2:
# systemctl -l status vdsmd
vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
Active: active (running) since Fri 2015-11-06 10:40:36 EST; 3h 55min ago
Process: 5035 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
Main PID: 6427 (vdsm)
CGroup: /system.slice/vdsmd.service
├─ 6334 /usr/libexec/ioprocess --read-pipe-fd 39
--write-pipe-fd 38 --max-threads 10 --max-queued-requests 10
├─ 6342 /usr/libexec/ioprocess --read-pipe-fd 48
--write-pipe-fd 46 --max-threads 10 --max-queued-requests 10
├─ 6354 /usr/libexec/ioprocess --read-pipe-fd 55
--write-pipe-fd 54 --max-threads 10 --max-queued-requests 10
├─ 6427 /usr/bin/python /usr/share/vdsm/vdsm
├─15071 /usr/libexec/ioprocess --read-pipe-fd 65
--write-pipe-fd 63 --max-threads 10 --max-queued-requests 10
├─15125 /usr/libexec/ioprocess --read-pipe-fd 73
--write-pipe-fd 72 --max-threads 10 --max-queued-requests 10
└─15140 /usr/libexec/ioprocess --read-pipe-fd 83
--write-pipe-fd 82 --max-threads 10 --max-queued-requests 10
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5
parse_server_challenge()
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5 ask_user_info()
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5
make_client_response()
Nov 06 10:40:37 virt2.eecs.yorku.ca python[6427]: DIGEST-MD5 client step 3
Nov 06 10:42:34 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.com.redhat.rhevm.vdsm
already removed
Nov 06 10:42:34 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8.org.qemu.guest_agent.0
already removed
Nov 06 10:43:56 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.com.redhat.rhevm.vdsm
already removed
Nov 06 10:43:56 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/77766cb4-0625-4432-a16f-def5e702102a.org.qemu.guest_agent.0
already removed
Nov 06 10:52:08 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.com.redhat.rhevm.vdsm
already removed
Nov 06 10:52:08 virt2.eecs.yorku.ca vdsm[6427]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/3bb83ad5-dc00-445d-8708-2db18c53e0e6.org.qemu.guest_agent.0
already removed
virt3:
# systemctl -l status vdsmd
vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
Active: active (running) since Fri 2015-11-06 10:06:28 EST; 4h 30min ago
Process: 12442 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh
--post-stop (code=exited, status=0/SUCCESS)
Process: 12446 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
Main PID: 12561 (vdsm)
CGroup: /system.slice/vdsmd.service
├─ 9066 /usr/libexec/ioprocess --read-pipe-fd 40
--write-pipe-fd 36 --max-threads 10 --max-queued-requests 10
├─ 9074 /usr/libexec/ioprocess --read-pipe-fd 48
--write-pipe-fd 47 --max-threads 10 --max-queued-requests 10
├─ 9085 /usr/libexec/ioprocess --read-pipe-fd 57
--write-pipe-fd 56 --max-threads 10 --max-queued-requests 10
├─12561 /usr/bin/python /usr/share/vdsm/vdsm
├─12726 /usr/libexec/ioprocess --read-pipe-fd 65
--write-pipe-fd 63 --max-threads 10 --max-queued-requests 10
├─12744 /usr/libexec/ioprocess --read-pipe-fd 72
--write-pipe-fd 74 --max-threads 10 --max-queued-requests 10
└─12750 /usr/libexec/ioprocess --read-pipe-fd 83
--write-pipe-fd 82 --max-threads 10 --max-queued-requests 10
Nov 06 10:09:15 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.com.redhat.rhevm.vdsm
already removed
Nov 06 10:09:15 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.org.qemu.guest_agent.0
already removed
Nov 06 10:10:17 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.com.redhat.rhevm.vdsm
already removed
Nov 06 10:10:17 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/aa487207-7ff4-465a-9d9b-2a103d50dc77.org.qemu.guest_agent.0
already removed
Nov 06 10:15:05 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/d66235b4-7c8d-4da2-a572-82ecc9c53a59.com.redhat.rhevm.vdsm
already removed
Nov 06 10:15:05 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/d66235b4-7c8d-4da2-a572-82ecc9c53a59.org.qemu.guest_agent.0
already removed
Nov 06 10:19:13 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/c64d80f9-a3bd-44f3-89c4-f1b9b509cdde.com.redhat.rhevm.vdsm
already removed
Nov 06 10:19:13 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/c64d80f9-a3bd-44f3-89c4-f1b9b509cdde.org.qemu.guest_agent.0
already removed
Nov 06 10:42:56 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.com.redhat.rhevm.vdsm
already removed
Nov 06 10:42:56 virt3.eecs.yorku.ca vdsm[12561]: vdsm root WARNING File:
/var/lib/libvirt/qemu/channels/62ff4ada-ee98-491e-bfb5-7adda7b513ee.org.qemu.guest_agent.0
already removed
--- I don't know what those warnings mean...
Jason.
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
