Problem solved.
I used Ansible and ran 'grep Duplicate /var/log/vdsm/vdsm.log' on all 121 of my
hosts. Only the log of the host currently running the hosted-engine had
entries.
I migrated the hosted-engine, and all other VMs to other hosts, and then put
the affected host into maintenance. I then undeployed it; then patched it; and
finally deployed it.
Its back up and running and taking VMs, and no more 'Duplicate keys' errors.
-----Original Message-----
From: matthew.st...@fujitsu.com <matthew.st...@fujitsu.com>
Sent: Monday, April 26, 2021 7:18 PM
To: Nir Soffer <nsof...@redhat.com>
Cc: users@ovirt.org
Subject: [ovirt-users] Re: Attempting to detach a storage domain
There is no /var/log/vdsm directory on the hosted-engine.
This is from the current host, hosting the engine: aaa.bbb.ccc.100 is the
hosted-engine
2021-04-26 19:12:42,760-0500 INFO (jsonrpc/2) [api.host] START
getCapabilities() from=::ffff:aaa.bbb.ccc.100,37812 (api:48)
2021-04-26 19:12:42,763-0500 INFO (jsonrpc/2) [api.host] FINISH
getCapabilities error=internal error: Duplicate key
from=::ffff:168.127.151.100,37812 (api:52)
2021-04-26 19:12:42,763-0500 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal
server error (__init__:350) Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
_handle_request
res = method(**params)
File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
_dynamicMethod
result = fn(*methodArgs)
File "<string>", line 2, in getCapabilities
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1371, in
getCapabilities
c = caps.get()
File "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 95, in get
machinetype.compatible_cpu_models())
File "/usr/lib/python2.7/site-packages/vdsm/common/cache.py", line 43, in
__call__
value = self.func(*args)
File "/usr/lib/python2.7/site-packages/vdsm/machinetype.py", line 142, in
compatible_cpu_models
all_models = domain_cpu_models(c, arch, cpu_mode)
File "/usr/lib/python2.7/site-packages/vdsm/machinetype.py", line 97, in
domain_cpu_models
domcaps = conn.getDomainCapabilities(None, arch, None, virt_type, 0)
File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py",
line 131, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in
wrapper
return func(inst, *args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3844, in
getDomainCapabilities
if ret is None: raise libvirtError ('virConnectGetDomainCapabilities()
failed', conn=self)
libvirtError: internal error: Duplicate key
2021-04-26 19:12:42,764-0500 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call
Host.getCapabilities failed (error -32603) in 0.00 seconds (__init__:312)
-----Original Message-----
From: Nir Soffer <nsof...@redhat.com>
Sent: Monday, April 26, 2021 5:12 PM
To: Stier, Matthew <matthew.st...@fujitsu.com>
Cc: Benny Zlotnik <bzlot...@redhat.com>; users@ovirt.org
Subject: Re: [ovirt-users] Re: Attempting to detach a storage domain
On Tue, Apr 27, 2021 at 12:48 AM matthew.st...@fujitsu.com
<matthew.st...@fujitsu.com> wrote:
>
> I'm getting tons of these in the
> hosted-engine:/var/log/ovirt-engine/engine.log
>
> 2021-04-26 16:40:40,260-05 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (EE-ManagedThreadFactory-engineScheduled-Thread-8) [] EVENT_ID:
> VDS_BROKER_COMMAND_FAILURE(10,802), VDSM xxxx.yyyy.zzz command Get
> Host Capabilities failed: Internal JSON-RPC error: {'reason':
> 'internal error: Duplicate key'}
This looks like bad response from vdsm. Can you share the vdsm log from the
host xxxx.yyyy.zzz?
/var/log/vdsm/vdsm.log
Nir
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org Privacy Statement:
https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z2GOGEZ2DZX34ADZ5L2BATIINYGVWYJD/
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SR23RTBYNLDO4BFYICPADGZLCM325JMN/
