[ovirt-users] Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version
Hi Strahil, I can't say it 100 % sure, but I think yes. We don't use the api a lot, but we have a running job to get all VM data regularly. There is no gap there. You can also work with the UI, but it takes extremely long to load the VMs UI and system load increases a lot. The engine I had this issue, has 8 vCPUs and load increased to 10 or 12 and stayed there for several minutes. When you then close the UI and wait some time, the load goes down to normal, about ~ 0.4. Now, after power cycling all VMs, the engine is fast like before the upgrade. What I'm really afraid of, is, what happened, when the engine has too much load and it looses the connectivity to their nodes. Will it then restart all of them? Probably yes and this would take down the whole environment and we will have a huge outage. And yes, we are a 24/7 company, where it is not so easy to reboot all VMs immediately. BR Florian - Ursprüngliche Mail - Von: "Strahil Nikolov" An: "users" , "Florian Schmid" Gesendet: Mittwoch, 26. Februar 2020 16:38:05 Betreff: Re: [ovirt-users] Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version On February 26, 2020 4:05:22 PM GMT+02:00, Florian Schmid wrote: >Good afternoon, > >after rebooting all the VMs, the CPU load of the engine is ok now. > >I think this high CPU usage is coming from this new feature, that you >see on a VM with orange triangle, what will be changed after power >cycle. >Actually, this is a nice feature, but it brings down an engine, when >all VMs need a reboot. After an upgrade for example. > >Is there a possibility to avoid this? How should I upgrade an oVirt >environment with 1000s of VMs? The engine would be unusable. > >BR Florian Hi Florian, Does the API work during this high load status ? Best Regards, Strahil Nikolov ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EPFSQ24VR2KSX5VQAEYZ6BDUF3CSRGMG/
[ovirt-users] Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version
On February 26, 2020 4:05:22 PM GMT+02:00, Florian Schmid wrote: >Good afternoon, > >after rebooting all the VMs, the CPU load of the engine is ok now. > >I think this high CPU usage is coming from this new feature, that you >see on a VM with orange triangle, what will be changed after power >cycle. >Actually, this is a nice feature, but it brings down an engine, when >all VMs need a reboot. After an upgrade for example. > >Is there a possibility to avoid this? How should I upgrade an oVirt >environment with 1000s of VMs? The engine would be unusable. > >BR Florian > > > > >Von: "Florian Schmid" >An: "users" >Gesendet: Freitag, 7. Februar 2020 08:13:05 >Betreff: [ovirt-users] Re: Huge increase of CPU load on hosted engine >VM after upgrade to 4.3.8 and cluster compatibility version > >Hi, > >no, fortunately, the bug didn't came back. > >It looks like, that the engine needs its time to settle after changing >the cluster compatibility version. After about 2 hours, the CPU load >was as low as before. >The time to load the VM UI takes a little bit more time, but I think >this is also normal, because all VMs still have the orange triangle and >need a reboot. > >Sorry for the false positive, but I was really afraid about the high >load. > >BR Florian > > >Von: "p staniforth" >An: "Florian Schmid" , "users" >Gesendet: Donnerstag, 6. Februar 2020 18:22:51 >Betreff: Re: Huge increase of CPU load on hosted engine VM after >upgrade to 4.3.8 and cluster compatibility version > >Has this bug come back > >engine.log flooded with "Field 'foo' can not be updated when status is >'Up'" > >[ >https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUHF7URJTQG753MND6PJ/?sort=date >| >https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUHF7URJTQG753MND6PJ/?sort=date >] > >Regards, > >Paul S. > >From: Florian Schmid >Sent: 06 February 2020 15:53 >To: users >Subject: [ovirt-users] Huge increase of CPU load on hosted engine VM >after upgrade to 4.3.8 and cluster compatibility version >Caution External Mail: Do not click any links or open any attachments >unless you trust the sender and know that the content is safe. > >Hi, > >I have upgraded yesterday one of our ovirt environments from 4.2.8 to >4.3.8. >We have a hosted engine, 25 hosts and about 150 VMs running there. > >After upgrading the cluster compatibility version from 4.2 to 4.3, I >have seen a huge increase of CPU load on the engine VM. >The engine had 4 cores (16GB mem) and never had load issues, but after >the upgrade, the system has a load of 6 and more. I have increased the >CPUs to 8, because working in the UI was impossible. > >I have noticed, that the load always increase, when I load the VM UI >tab, where now all VMs have the orange triangle with pending changes: >Custom compatibility version > >The problem I have, is, that I can't reboot now all VMs, maybe in the >next 3 or 4 weeks. >Working now with the UI is quite problematic, because of the slowness. > >The engine VM was already rebooted, but didn't solve the problem. > >What logs do you need and is there a workaround available? I have some >more oVirt environments to upgrade, all with even more VMs. > >BR Florian >___ >Users mailing list -- users@ovirt.org >To unsubscribe send an email to users-le...@ovirt.org >Privacy Statement: [ >https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fsite%2Fprivacy-policy%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074511312&sdata=wQAhPwaKrYVwjTorzE0CKjs43uPjGuyJX8T0ceJwMBc%3D&reserved=0 >| >https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fsite%2Fprivacy-policy%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074511312&sdata=wQAhPwaKrYVwjTorzE0CKjs43uPjGuyJX8T0ceJwMBc%3D&reserved=0 >] >oVirt Code of Conduct: [ >https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074516304&sdata=n6GIoSEuBbfNYLfqMvJD5aszsSV0vftLMdOtStg3hFo%3D&reserved=0 >| >https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fcomm
[ovirt-users] Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version
Good afternoon, after rebooting all the VMs, the CPU load of the engine is ok now. I think this high CPU usage is coming from this new feature, that you see on a VM with orange triangle, what will be changed after power cycle. Actually, this is a nice feature, but it brings down an engine, when all VMs need a reboot. After an upgrade for example. Is there a possibility to avoid this? How should I upgrade an oVirt environment with 1000s of VMs? The engine would be unusable. BR Florian Von: "Florian Schmid" An: "users" Gesendet: Freitag, 7. Februar 2020 08:13:05 Betreff: [ovirt-users] Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Hi, no, fortunately, the bug didn't came back. It looks like, that the engine needs its time to settle after changing the cluster compatibility version. After about 2 hours, the CPU load was as low as before. The time to load the VM UI takes a little bit more time, but I think this is also normal, because all VMs still have the orange triangle and need a reboot. Sorry for the false positive, but I was really afraid about the high load. BR Florian Von: "p staniforth" An: "Florian Schmid" , "users" Gesendet: Donnerstag, 6. Februar 2020 18:22:51 Betreff: Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Has this bug come back engine.log flooded with "Field 'foo' can not be updated when status is 'Up'" [ https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUHF7URJTQG753MND6PJ/?sort=date | https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUHF7URJTQG753MND6PJ/?sort=date ] Regards, Paul S. From: Florian Schmid Sent: 06 February 2020 15:53 To: users Subject: [ovirt-users] Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. Hi, I have upgraded yesterday one of our ovirt environments from 4.2.8 to 4.3.8. We have a hosted engine, 25 hosts and about 150 VMs running there. After upgrading the cluster compatibility version from 4.2 to 4.3, I have seen a huge increase of CPU load on the engine VM. The engine had 4 cores (16GB mem) and never had load issues, but after the upgrade, the system has a load of 6 and more. I have increased the CPUs to 8, because working in the UI was impossible. I have noticed, that the load always increase, when I load the VM UI tab, where now all VMs have the orange triangle with pending changes: Custom compatibility version The problem I have, is, that I can't reboot now all VMs, maybe in the next 3 or 4 weeks. Working now with the UI is quite problematic, because of the slowness. The engine VM was already rebooted, but didn't solve the problem. What logs do you need and is there a workaround available? I have some more oVirt environments to upgrade, all with even more VMs. BR Florian ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fsite%2Fprivacy-policy%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074511312&sdata=wQAhPwaKrYVwjTorzE0CKjs43uPjGuyJX8T0ceJwMBc%3D&reserved=0 | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fsite%2Fprivacy-policy%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074511312&sdata=wQAhPwaKrYVwjTorzE0CKjs43uPjGuyJX8T0ceJwMBc%3D&reserved=0 ] oVirt Code of Conduct: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074516304&sdata=n6GIoSEuBbfNYLfqMvJD5aszsSV0vftLMdOtStg3hFo%3D&reserved=0 | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074516304&sdata=n6GIoSEuBbfNYLfqMvJD5aszsSV0vftLMdOtStg3hFo%3D&reserved=0 ] List Archives: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovirt.org%2Farchives%2Flist%2Fusers%40ovirt.org%2Fmessage%2FOQYXCKCJGCDP6EJTXV5T6Z63SIQYEVXD%2F&data=02%7C01%7Cp.stan
[ovirt-users] Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version
Hi, no, fortunately, the bug didn't came back. It looks like, that the engine needs its time to settle after changing the cluster compatibility version. After about 2 hours, the CPU load was as low as before. The time to load the VM UI takes a little bit more time, but I think this is also normal, because all VMs still have the orange triangle and need a reboot. Sorry for the false positive, but I was really afraid about the high load. BR Florian Von: "p staniforth" An: "Florian Schmid" , "users" Gesendet: Donnerstag, 6. Februar 2020 18:22:51 Betreff: Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Has this bug come back engine.log flooded with "Field 'foo' can not be updated when status is 'Up'" [ https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUHF7URJTQG753MND6PJ/?sort=date | https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUHF7URJTQG753MND6PJ/?sort=date ] Regards, Paul S. From: Florian Schmid Sent: 06 February 2020 15:53 To: users Subject: [ovirt-users] Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. Hi, I have upgraded yesterday one of our ovirt environments from 4.2.8 to 4.3.8. We have a hosted engine, 25 hosts and about 150 VMs running there. After upgrading the cluster compatibility version from 4.2 to 4.3, I have seen a huge increase of CPU load on the engine VM. The engine had 4 cores (16GB mem) and never had load issues, but after the upgrade, the system has a load of 6 and more. I have increased the CPUs to 8, because working in the UI was impossible. I have noticed, that the load always increase, when I load the VM UI tab, where now all VMs have the orange triangle with pending changes: Custom compatibility version The problem I have, is, that I can't reboot now all VMs, maybe in the next 3 or 4 weeks. Working now with the UI is quite problematic, because of the slowness. The engine VM was already rebooted, but didn't solve the problem. What logs do you need and is there a workaround available? I have some more oVirt environments to upgrade, all with even more VMs. BR Florian ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fsite%2Fprivacy-policy%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074511312&sdata=wQAhPwaKrYVwjTorzE0CKjs43uPjGuyJX8T0ceJwMBc%3D&reserved=0 | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fsite%2Fprivacy-policy%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074511312&sdata=wQAhPwaKrYVwjTorzE0CKjs43uPjGuyJX8T0ceJwMBc%3D&reserved=0 ] oVirt Code of Conduct: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074516304&sdata=n6GIoSEuBbfNYLfqMvJD5aszsSV0vftLMdOtStg3hFo%3D&reserved=0 | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074516304&sdata=n6GIoSEuBbfNYLfqMvJD5aszsSV0vftLMdOtStg3hFo%3D&reserved=0 ] List Archives: [ https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovirt.org%2Farchives%2Flist%2Fusers%40ovirt.org%2Fmessage%2FOQYXCKCJGCDP6EJTXV5T6Z63SIQYEVXD%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074516304&sdata=%2B1GDKaTB2TpU%2F0Y7bWMyZhLOotk3knBwk%2FjMqipZ9Kg%3D&reserved=0 | https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovirt.org%2Farchives%2Flist%2Fusers%40ovirt.org%2Fmessage%2FOQYXCKCJGCDP6EJTXV5T6Z63SIQYEVXD%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074516304&sdata=%2B1GDKaTB2TpU%2F0Y7bWMyZhLOotk3knBwk%2FjMqipZ9Kg%3D&reserved=0 ] To view the terms under which this email is distributed, please go to:- [ http://leedsbeckett.ac.uk/disclaimer/email/ | http://leedsbeckett.ac.uk/disclaimer/email/ ] ___ Users mailing list -- users@ovirt.org To unsubscribe send an email t
[ovirt-users] Re: Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version
Has this bug come back engine.log flooded with "Field 'foo' can not be updated when status is 'Up'" https://lists.ovirt.org/archives/list/users@ovirt.org/thread/MKFQRCKHRT6NJUHF7URJTQG753MND6PJ/?sort=date Regards, Paul S. From: Florian Schmid Sent: 06 February 2020 15:53 To: users Subject: [ovirt-users] Huge increase of CPU load on hosted engine VM after upgrade to 4.3.8 and cluster compatibility version Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. Hi, I have upgraded yesterday one of our ovirt environments from 4.2.8 to 4.3.8. We have a hosted engine, 25 hosts and about 150 VMs running there. After upgrading the cluster compatibility version from 4.2 to 4.3, I have seen a huge increase of CPU load on the engine VM. The engine had 4 cores (16GB mem) and never had load issues, but after the upgrade, the system has a load of 6 and more. I have increased the CPUs to 8, because working in the UI was impossible. I have noticed, that the load always increase, when I load the VM UI tab, where now all VMs have the orange triangle with pending changes: Custom compatibility version The problem I have, is, that I can't reboot now all VMs, maybe in the next 3 or 4 weeks. Working now with the UI is quite problematic, because of the slowness. The engine VM was already rebooted, but didn't solve the problem. What logs do you need and is there a workaround available? I have some more oVirt environments to upgrade, all with even more VMs. BR Florian ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fsite%2Fprivacy-policy%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074511312&sdata=wQAhPwaKrYVwjTorzE0CKjs43uPjGuyJX8T0ceJwMBc%3D&reserved=0 oVirt Code of Conduct: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074516304&sdata=n6GIoSEuBbfNYLfqMvJD5aszsSV0vftLMdOtStg3hFo%3D&reserved=0 List Archives: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovirt.org%2Farchives%2Flist%2Fusers%40ovirt.org%2Fmessage%2FOQYXCKCJGCDP6EJTXV5T6Z63SIQYEVXD%2F&data=02%7C01%7Cp.staniforth%40leedsbeckett.ac.uk%7C15e021512d1e4471aa8608d7ab1e90c1%7Cd79a81124fbe417aa112cd0fb490d85c%7C0%7C0%7C637166020074516304&sdata=%2B1GDKaTB2TpU%2F0Y7bWMyZhLOotk3knBwk%2FjMqipZ9Kg%3D&reserved=0 To view the terms under which this email is distributed, please go to:- http://leedsbeckett.ac.uk/disclaimer/email/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/5KA7OY56W2RZ65M4NIQH6W3ZAJGD6MVB/
