Hi Nir,
Thank you very much for all the help and information.
We will continue to investigate the NFS server side. To find what may be
causing one of the hosts to lose access to storage.
The strange thing is that it happens only on one of the NFS client hosts
and not on all of them at the same time.
Pablo.
El 17/02/2022 a las 11:02, Nir Soffer escribió:
On Thu, Feb 17, 2022 at 11:58 AM Nir Soffer <nsof...@redhat.com> wrote:
On Thu, Feb 17, 2022 at 11:20 AM Pablo Olivera <p.oliv...@telfy.com> wrote:
Hi Nir,
Thank you very much for your detailed explanations.
The pid 6398 looks like it's HostedEngine:
audit/audit.log:type=VIRT_CONTROL msg=audit(1644587639.935:7895): pid=3629 uid=0 auid=4294967295 ses=4294967295
subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=start reason=booted vm="HostedEngine"
uuid=37a75c8e-50a2-4abd-a887-8a62a75814cc vm-pid=6398 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=?
res=success'UID="root" AUID="unset"
So, I understand that SanLock has problems with the storage (it loses
connection with NFS storage). The watchdog begins to check connectivity with
the MV and after the established time, the order to
reboot the machine.
I don't know if I can somehow increase these timeouts, or try to make sanlock
force the reconnection or renewal with the storage and in this way try to avoid
host reboots for this reason.
You can do one of these:
1. Use lower timeouts on the NFS server mount, so the NFS server at
the same time
the sanlock lease times out.
2. Use larger sanlock timeout so sanlock lease time out when the NFS
server times out.
3. Both 1 and 2
The problem is that NFS timeouts are not predictable. In the past we used:
"timeo=600,retrans=6" which can lead to 21 minutes timeout, but practically
we saw up to a 30 minutes timeout.
In https://github.com/oVirt/vdsm/commit/672a98bbf3e55d1077669f06c37305185fbdc289
we change this to the recommended seting:
"timeo=100,retrans=3"
Which according to the docs, should fail in 60 seconds if all retries
fail. But practically we
saw up to 270 seconds timeout with this setting, which does not play
well with sanlock.
We assumed that the timeout value should not be less than sanlock io timeout
(10 seconds) but I'm not sure this assumption is correct.
You can smaller timeout value in engine storage domain
"custom connections parameters"
- Retransmissions - mapped to "retrans" mount option
- Timeout (deciseconds) - mapped to "timeo" mount option
For example:
Retransmissions: 3
Timeout: 5
Correction:
Timeout: 50 (5 seconds, 50 deciseconds)
Theoretically this will behave like this:
00:00 retry 1 (5 seconds timeout)
00:10 retry 2 (10 seconds timeout)
00:30 retry 3 (15 seconds timeout)
00:45 request fail
But based on what we see with the defaults, this is likely to take more time.
If it fails before 140 seconds, the VM will be killed and the host
will not reboot.
The other way is to increase sanlock timeout, in vdsm configuration.
note that changing sanlock timeout requires also changing other
settings (e.g. spm:watchdog_interval).
Add this file on all hosts:
$ cat /etc/vdsm/vdsm.conf.d/99-local.conf
[spm]
# If enabled, montior the SPM lease status and panic if the lease
# status is not expected. The SPM host will lose the SPM role, and
# engine will select a new SPM host. (default true)
# watchdog_enable = true
# Watchdog check internal in seconds. The recommended value is
# sanlock:io_timeout * 2. (default 20)
watchdog_interval = 40
[sanlock]
# I/O timeout in seconds. All sanlock timeouts are computed based on
# this value. Using larger timeout will make VMs more resilient to
# short storage outage, but increase VM failover time and the time to
# acquire a host id. For more info on sanlock timeouts please check
# sanlock source:
# https://pagure.io/sanlock/raw/master/f/src/timeouts.h. If your
# storage requires larger timeouts, you can increase the value to 15
# or 20 seconds. If you change this you need to update also multipath
# no_path_retry. For more info onconfiguring multipath please check
# /etc/multipath.conf.oVirt is tested only with the default value (10
# seconds)
io_timeout = 20
You can check https://github.com/oVirt/vdsm/blob/master/doc/io-timeouts.md
to learn more about sanlock timeouts.
Alternatively, you can make a small change in NFS timeout and small change in
sanlock timeout to make them work better together.
All this is of course to handle the case when the NFS server is not accessible,
but this is something that should not happen in a healthy cluster. You need
to check why the server was not accessible and fix this problem.
Nir
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KUBSSNLNF2LAJ2LDNE2SKZJOA6SY55G7/
