Re: [CentOS-virt] very low performance of Xen guests
On 15-06-2020 23:42, Manuel Wolfshant wrote: On 6/15/20 2:46 PM, Stephen John Smoogen wrote: On Sun, 14 Jun 2020 at 14:49, Manuel Wolfshant wrote: Hello For the past months I've been testing upgrading my Xen hosts to CentOS 7 and I face an issue for which I need your help to solve. The testing machines are IBM blades, model H21 and H21XM. Initial tests were performed on the H21 with 16 GB RAM; during the last 6=7 weeks I've been using the H21XM with 64 GB. In all cases the guests were fully updated CentOS 7 -- initially 7.6 ( most recent at the time of the initial tests ), and respectively 7.8 for the tests performed during the last 2 months. As host I used initially CentOS 6 with latest kernel available in the centos virt repo at the time of the tests and CentOS 7 with the latest kernel as well. As xen versions I tested 4.8 and 4.12 ( xl info included below ). The storage for the last tests is a Crucial MX500 but results were similar when using traditional HDD. My problem, in short, is that the guests are extremely slow. For instance , in the most recent tests, a yum install kernel takes cca 1 min on the host and 12-15 (!!!) minutes in the guest, all time being spent in dracut regenerating the initramfs images. I've done rough tests with the storage ( via dd if=/dev/zero of=a_test_file size bs=10M count=1000 ) and the speed was comparable between the hosts and the guests. The version of the kernel in use inside the guest also did not seem to make any difference . OTOH, sysbench ( https://github.com/akopytov/sysbench/ ) as well as p7zip benchmark report for the guests a speed which is between 10% and 50% of the host. Quite obviously, changing the elevator had no influence either. Here is the info which I think that should be relevant for the software versions in use. Feel free to ask for any additional info. Is there a way to boot up a PV guest versus an HVM? If I understood the docs correctly, newer xen does only PVHVM ( xen_platform_pci=1 activates that ) and HVM. But they say it's better than PV. And I did verify, PVHVM is indeed enabled and active You can also test running your Linux domUs as Xen PVH [1], which requires kernel 4.11 and Xen 4.10 as a minimum. [1] https://wiki.xenproject.org/wiki/Linux_PVH -- Adi Pircalabu ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] very low performance of Xen guests
On 6/15/20 2:46 PM, Stephen John Smoogen wrote: On Sun, 14 Jun 2020 at 14:49, Manuel Wolfshant mailto:wo...@nobugconsulting.ro>> wrote: Hello For the past months I've been testing upgrading my Xen hosts to CentOS 7 and I face an issue for which I need your help to solve. The testing machines are IBM blades, model H21 and H21XM. Initial tests were performed on the H21 with 16 GB RAM; during the last 6=7 weeks I've been using the H21XM with 64 GB. In all cases the guests were fully updated CentOS 7 -- initially 7.6 ( most recent at the time of the initial tests ), and respectively 7.8 for the tests performed during the last 2 months. As host I used initially CentOS 6 with latest kernel available in the centos virt repo at the time of the tests and CentOS 7 with the latest kernel as well. As xen versions I tested 4.8 and 4.12 ( xl info included below ). The storage for the last tests is a Crucial MX500 but results were similar when using traditional HDD. My problem, in short, is that the guests are extremely slow. For instance , in the most recent tests, a yum install kernel takes cca 1 min on the host and 12-15 (!!!) minutes in the guest, all time being spent in dracut regenerating the initramfs images. I've done rough tests with the storage ( via dd if=/dev/zero of=a_test_file size bs=10M count=1000 ) and the speed was comparable between the hosts and the guests. The version of the kernel in use inside the guest also did not seem to make any difference . OTOH, sysbench ( https://github.com/akopytov/sysbench/ ) as well as p7zip benchmark report for the guests a speed which is between 10% and 50% of the host. Quite obviously, changing the elevator had no influence either. Here is the info which I think that should be relevant for the software versions in use. Feel free to ask for any additional info. Is there a way to boot up a PV guest versus an HVM? If I understood the docs correctly, newer xen does only PVHVM ( xen_platform_pci=1 activates that ) and HVM. But they say it's better than PV. And I did verify, PVHVM is indeed enabled and active I could not find a H21XM but found an HS21XM on the iBM My bad. The blades are indeed HS21 (Type 8853) and HS21 XM (Type 7995). The XM blades have 2*Xeon E5450@3GHz / 12GB L1 cache processors. The options I can fiddle with are https://imgur.com/a/DonXe5P AFAICS the setttings are reasonable but please do let me know if there is anything there that should not be as it is site and that seemed to be a 4 core 8 thread cpu which looks 'old' enough that the Spectre/etc fixes to improve performance after the initial hit were not done. (Basically I was told that if the CPU was older than 2012, just turn off hyperthreading altogether to try and get back some performance.. but don't expect much). I can live with that. My problem is that DomU are much much slower that Dom0 so it seems xen virtualization affects ( heavily ) the performance. As such I would also try turning off HT on the CPU to see if that improves anything. I got inspired by Adi's earlier suggestion and after reading https://access.redhat.com/articles/3311301 I've tried today all variants of disabling the spectre mitigations. Whatever I do, immediately after a reboot, yum reinstall kernel does not take less than 5 minutes :( It goes down to 2 min if I repeat the operation afterwards so I guess some caching kicks in. I will try later today the kernels from elrepo and maybe even xen.crc.id.au ( I kind of hate the "disable selinux" recommendation from the install page so I postponed it in the hope of other solution ). ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] very low performance of Xen guests
On Sun, 14 Jun 2020 at 14:49, Manuel Wolfshant wrote: > Hello > > > For the past months I've been testing upgrading my Xen hosts to CentOS > 7 and I face an issue for which I need your help to solve. > > The testing machines are IBM blades, model H21 and H21XM. Initial > tests were performed on the H21 with 16 GB RAM; during the last 6=7 weeks > I've been using the H21XM with 64 GB. In all cases the guests were fully > updated CentOS 7 -- initially 7.6 ( most recent at the time of the initial > tests ), and respectively 7.8 for the tests performed during the last 2 > months. As host I used initially CentOS 6 with latest kernel available in > the centos virt repo at the time of the tests and CentOS 7 with the latest > kernel as well. As xen versions I tested 4.8 and 4.12 ( xl info included > below ). The storage for the last tests is a Crucial MX500 but results were > similar when using traditional HDD. > > My problem, in short, is that the guests are extremely slow. For > instance , in the most recent tests, a yum install kernel takes cca 1 min > on the host and 12-15 (!!!) minutes in the guest, all time being spent in > dracut regenerating the initramfs images. I've done rough tests with the > storage ( via dd if=/dev/zero of=a_test_file size bs=10M count=1000 ) and > the speed was comparable between the hosts and the guests. The version of > the kernel in use inside the guest also did not seem to make any difference > . OTOH, sysbench ( https://github.com/akopytov/sysbench/ ) as well as > p7zip benchmark report for the guests a speed which is between 10% and 50% > of the host. Quite obviously, changing the elevator had no influence > either. > > Here is the info which I think that should be relevant for the > software versions in use. Feel free to ask for any additional info. > Is there a way to boot up a PV guest versus an HVM? I could not find a H21XM but found an HS21XM on the iBM site and that seemed to be a 4 core 8 thread cpu which looks 'old' enough that the Spectre/etc fixes to improve performance after the initial hit were not done. (Basically I was told that if the CPU was older than 2012, just turn off hyperthreading altogether to try and get back some performance.. but don't expect much). As such I would also try turning off HT on the CPU to see if that improves anything. -- Stephen J Smoogen. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
