[ovirt-users] Problem with ISCSI multipathing identifying faild path to a LUN every few seconds.
Hi All, In the past few days, we have been having problems with hosts becoming unmanageable due to multipathd identifying false failed paths and VDSM crashing because of it. We were running version 4.4.1 and upgrading to 4.4.5(engine) and 4.4.6(nodes) seems to have resolved the VDSM. However, currently, we see that the multipathing events continue. From what we have observed, the events start in correlation to the host reporting on low swap space. The low swap space seems to be related to Commvault backup operation running. By running top on the host while a backup operation is running I can see swap being consumed to 100% although there is plenty of RAM available. After the multipath events start happening the only means of stopping it was to reboot the host. This is the warning in oVirt UI: Apr 24, 2021, 9:14:07 PM - Available swap memory of host Ovirt-Node2 [953 MB] is under defined threshold [1024 MB]. This is the first appearance of the multipath event in /var/log/messages: Apr 24 21:14:58 ovirt-node2 kernel: sd 21:0:0:9: [sdgi] tag#77 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=15s Apr 24 21:14:58 ovirt-node2 kernel: sd 21:0:0:9: [sdgi] tag#77 Sense Key : Aborted Command [current] Apr 24 21:14:58 ovirt-node2 kernel: sd 21:0:0:9: [sdgi] tag#77 <>ASC=0xc1 ASCQ=0x1 Apr 24 21:14:58 ovirt-node2 kernel: sd 21:0:0:9: [sdgi] tag#77 CDB: Read(16) 88 00 00 00 00 00 79 ac 75 a0 00 00 06 00 00 00 Apr 24 21:14:58 ovirt-node2 kernel: blk_update_request: I/O error, dev sdgi, sector 2041345440 op 0x0:(READ) flags 0x4200 phys_seg 192 prio class 0 Apr 24 21:14:58 ovirt-node2 kernel: device-mapper: multipath: 253:20: Failing path 131:224. Apr 24 21:14:58 ovirt-node2 multipathd[3044]: sdgi: mark as failed Apr 24 21:14:58 ovirt-node2 multipathd[3044]: 360e00d2c002cb4a8000b: remaining active paths: 11 Apr 24 21:15:03 ovirt-node2 multipathd[3044]: 360e00d2c002cb4a8000b: sdgi - tur checker reports path is up Apr 24 21:15:03 ovirt-node2 multipathd[3044]: 131:224: reinstated Apr 24 21:15:03 ovirt-node2 multipathd[3044]: 360e00d2c002cb4a8000b: remaining active paths: 12 Apr 24 21:15:03 ovirt-node2 kernel: device-mapper: multipath: 253:20: Reinstating path 131:224. Apr 24 21:15:03 ovirt-node2 kernel: sd 21:0:0:9: alua: port group 8091 state A preferred supports toluSNA Apr 24 21:15:03 ovirt-node2 kernel: sd 21:0:0:9: alua: port group 8091 state A preferred supports toluSNA Apr 24 21:15:13 ovirt-node2 kernel: sd 13:0:0:9: [sdau] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=15s Apr 24 21:15:13 ovirt-node2 kernel: sd 13:0:0:9: [sdau] tag#25 Sense Key : Aborted Command [current] Apr 24 21:15:13 ovirt-node2 kernel: sd 13:0:0:9: [sdau] tag#25 <>ASC=0xc1 ASCQ=0x1 Apr 24 21:15:13 ovirt-node2 kernel: sd 13:0:0:9: [sdau] tag#25 CDB: Read(16) 88 00 00 00 00 00 79 ac 75 a0 00 00 06 00 00 00 Apr 24 21:15:13 ovirt-node2 kernel: blk_update_request: I/O error, dev sdau, sector 2041345440 op 0x0:(READ) flags 0x4200 phys_seg 192 prio class 0 Apr 24 21:15:13 ovirt-node2 kernel: device-mapper: multipath: 253:20: Failing path 66:224. Apr 24 21:15:13 ovirt-node2 multipathd[3044]: sdau: mark as failed Underlying storage is Fujitsu DX200 S5 with all SSD drives. Each host has two 10Gbit network adapters dedicated to ISCSI. Any help with this would be highly appreciated. Thanks, Gal Villaret ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/J2KX2FTUKIG3F6XEYSSSVJ7TSD3P4NW5/
[ovirt-users] Increase the initial size of LVM snapshots to prevent VM's from freezing
Hi all, Lately, I have been encountering an issue where VMs freeze during backups. From what I can gather, this happens because some of the VMs sometimes perform large writes during the backup window and the snapshots dose not grow fast enough. I use ISCSI storage with all VM disks preallocated. Is there a configuration value I can change in order to increase the initial size of snapshots and also maybe change the watermark trigger for the expansion of snapshots? Thanks. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ELN2Z42XHZJBWZZC6X2NSE3E63DGHM3K/
[ovirt-users] Re: VM Portal on a stand alone server
Thanks. Using a proxy is my plan. I was asked by my security team to add some more "distance" from users to the server running the engine and the database. Coming to think of it, maybe deploying the DB on a standalone server is a better approach. On Wed, Jul 15, 2020 at 10:08 PM Michal Skrivanek < michal.skriva...@redhat.com> wrote: > > > On 15 Jul 2020, at 17:12, Gal Villaret wrote: > > I have the engine running on a dedicated server. > > I would like to separate the VM Portal to another server or maybe have > another instance of VM portal on another server. > > The idea is to be able to put the VM Portal on a different subnet and to > put a firewall between it and the engine. > > > it’s possible in devel mode, see docker container at > https://github.com/oVirt/ovirt-web-ui > but we’re not really updating it very often, I wouldn’t recommend it for > production > > What’s the concern if you limit access to :443 and use a proxy? There’s > not much more a random joe can do without a permission to log into > webadmin, there’s just a static landing page with few links. > > > Thanks. > > On Wed, Jul 15, 2020, 17:18 Sandro Bonazzola wrote: > >> >> >> Il giorno mer 15 lug 2020 alle ore 12:26 ha >> scritto: >> >>> Hi Folks, >>> >>> I'm rather new to oVirt and loving it. >>> running 4.4.1. >>> I would like to be able to run the VM Portal on a stand-alone server for >>> security concerns. >>> >>> Can anyone point in the right direction for achieving this? >>> >> >> Can you please elaborate? >> Are you asking for having the admin portal and the VM user portal running >> on 2 different servers? >> Or running the engine on a dedicated server instead of on self hosted >> engine VM? >> >> >> >> >>> >>> Thanks, >>> >>> Gal Villaret >>> ___ >>> Users mailing list -- users@ovirt.org >>> To unsubscribe send an email to users-le...@ovirt.org >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/3BE6CNDNLDKZQW5ANOC3UFT3BQZZFGHC/ >>> >> >> >> -- >> Sandro Bonazzola >> MANAGER, SOFTWARE ENGINEERING, EMEA R RHV >> >> Red Hat EMEA <https://www.redhat.com/> >> sbona...@redhat.com >> <https://www.redhat.com/> >> >> *Red Hat respects your work life balance. Therefore there is no need to >> answer this email out of your office hours. >> <https://mojo.redhat.com/docs/DOC-1199578>* >> > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/EDCHL6QLGDWU2GASCOXHC4B3DFHJNPBC/ > > > ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/W7S2KFDTCRV3HDV5MYQQTPID2ZWMLLCX/
[ovirt-users] Re: VM Portal on a stand alone server
I have the engine running on a dedicated server. I would like to separate the VM Portal to another server or maybe have another instance of VM portal on another server. The idea is to be able to put the VM Portal on a different subnet and to put a firewall between it and the engine. Thanks. On Wed, Jul 15, 2020, 17:18 Sandro Bonazzola wrote: > > > Il giorno mer 15 lug 2020 alle ore 12:26 ha > scritto: > >> Hi Folks, >> >> I'm rather new to oVirt and loving it. >> running 4.4.1. >> I would like to be able to run the VM Portal on a stand-alone server for >> security concerns. >> >> Can anyone point in the right direction for achieving this? >> > > Can you please elaborate? > Are you asking for having the admin portal and the VM user portal running > on 2 different servers? > Or running the engine on a dedicated server instead of on self hosted > engine VM? > > > > >> >> Thanks, >> >> Gal Villaret >> ___ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-le...@ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/3BE6CNDNLDKZQW5ANOC3UFT3BQZZFGHC/ >> > > > -- > > Sandro Bonazzola > > MANAGER, SOFTWARE ENGINEERING, EMEA R RHV > > Red Hat EMEA <https://www.redhat.com/> > > sbona...@redhat.com > <https://www.redhat.com/> > > *Red Hat respects your work life balance. Therefore there is no need to > answer this email out of your office hours. > <https://mojo.redhat.com/docs/DOC-1199578>* > ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EDCHL6QLGDWU2GASCOXHC4B3DFHJNPBC/
[ovirt-users] VM Portal on a stand alone server
Hi Folks, I'm rather new to oVirt and loving it. running 4.4.1. I would like to be able to run the VM Portal on a stand-alone server for security concerns. Can anyone point in the right direction for achieving this? Thanks, Gal Villaret ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3BE6CNDNLDKZQW5ANOC3UFT3BQZZFGHC/
