Re: [ovirt-users] Guest VM Running 160 RCU Processes
On 10/21/2014 07:13 PM, Darrell Budic wrote: Was poking at this a little to see if there was any tuning that could affect it and spotted some oddness with the processor counts on my VMs under Ovirt 3.4. They seem to think they only have the proper number I set in ovirt (as shown in /proc/cpu), but if I look at dmidecode, there’s 159 bogus processors listed. I’d expect maybe 16 from the -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 argument to qemu-kvm, but there are 0xa0 of them. Maybe this is a seabios or qemu-kvm issue causing all those extras? Anyway, the # of rcu* processes matches pretty well, so that’s likely where it’s coming from. At least they shouldn’t be causing a performance issue, given their purpose as non-blocking work threads, but it is odd to see. Punit, did you open a BZ I can add these details to? BTW, this appears to be corrected with some component of Ovirt 3.5 (probably qemu-rhev?). On VMs started after I upgraded my Engine (even on 3.4 vdsmds), I’m only seeing 16 “processors” in the bios, and thus only 16 of the various rcu* processes. Could have been a general Centos 6.5 update too, since I did those as well, so I can’t get any finer resolution on that issue (both engine and host nodes, lots of Centos 7 VMs). If there is a RFE for this, perhaps a configurable max # of CPU sockets for hot add could be added, or it could be limited to the max physical cpu count of the biggest host in the cluster? there are configs for that already: /etc/ovirt-engine/engine-config/engine-config.properties:MaxNumOfVmCpus /etc/ovirt-engine/engine-config/engine-config.properties:MaxNumOfVmSockets /etc/ovirt-engine/engine-config/engine-config.properties:MaxNumOfCpuPerSocket -Darrell On Oct 20, 2014, at 7:34 AM, Doron Fediuck dfedi...@redhat.com wrote: - Original Message - From: Punit Dambiwal hypu...@gmail.com To: users@ovirt.org, Dan Kenigsberg dan...@redhat.com, Itamar Heim ih...@redhat.com, aha...@redhat.com Sent: Monday, October 20, 2014 5:58:20 AM Subject: Re: [ovirt-users] Guest VM Running 160 RCU Processes Hi, Is there any body suggest me good way to handle it?? On Fri, Oct 17, 2014 at 3:15 PM, Punit Dambiwal hypu...@gmail.com wrote: Hi, I have one Ovirt cluster and under this cluster all the guest machines (such as centos,ubuntu,debian etc) almost 160 RCU processes running i searched on google about the RCU (It's kernel process read-copy-upate) http://lwn.net/Articles/518953/ I want to know how i can modify this 160 processes to 10-20 or how i can to disable itis there any bed impact if i disable it Thanks, Punit Hi Punit, we need to do a bit of a research on this one. In order to make sure we keep track of it, do you mind opening an RFE (BZ) with all the relevant details, including the hardware that you're using, guests config, how busy the guests/host are, Which hypervisor is used and the versions. Thanks, Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Guest VM Running 160 RCU Processes
On 10/22/2014 02:43 PM, Michal Skrivanek wrote: On Oct 21, 2014, at 18:13 , Darrell Budic bu...@onholyground.com wrote: Was poking at this a little to see if there was any tuning that could affect it and spotted some oddness with the processor counts on my VMs under Ovirt 3.4. They seem to think they only have the proper number I set in ovirt (as shown in /proc/cpu), but if I look at dmidecode, there’s 159 bogus processors listed. I’d expect maybe 16 from the -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 argument to qemu-kvm, but there are 0xa0 of them. Maybe this is a seabios or qemu-kvm issue causing all those extras? Anyway, the # of rcu* processes matches pretty well, so that’s likely where it’s coming from. At least they shouldn’t be causing a performance issue, given their purpose as non-blocking work threads, but it is odd to see. Punit, did you open a BZ I can add these details to? BTW, this appears to be corrected with some component of Ovirt 3.5 (probably qemu-rhev?). On VMs started after I upgraded my Engine (even on 3.4 vdsmds), I’m only seeing 16 “processors” in the bios, and thus only 16 of the various rcu* processes. Could have been a general Centos 6.5 update too, since I did those as well, so I can’t get any finer resolution on that issue (both engine and host nodes, lots of Centos 7 VMs). If there is a RFE for this, perhaps a configurable max # of CPU sockets for hot add could be added, or it could be limited to the max physical cpu count of the biggest host in the cluster? The max is 16. Roy, what can we change/not change after your latest changes? I suppose it's related to teh maximum values we send because of hotplug support I guess its related. What is surprising is that probably most of the processors are offline, therefor not in use but the kernel still allocated rcu's to them. maybe a bug, maybe an optimization for online speed, don't know really. 2 config values would help engine-config -g MaxNumOfVmSockets engine-config -g MaxNumOfVmCpus You can still have 160 vcpu (that limit is QEMU) - 16 Sockets and 10 cpu p/socket Roy Thanks, michal -Darrell On Oct 20, 2014, at 7:34 AM, Doron Fediuck dfedi...@redhat.com wrote: - Original Message - From: Punit Dambiwal hypu...@gmail.com To: users@ovirt.org, Dan Kenigsberg dan...@redhat.com, Itamar Heim ih...@redhat.com, aha...@redhat.com Sent: Monday, October 20, 2014 5:58:20 AM Subject: Re: [ovirt-users] Guest VM Running 160 RCU Processes Hi, Is there any body suggest me good way to handle it?? On Fri, Oct 17, 2014 at 3:15 PM, Punit Dambiwal hypu...@gmail.com wrote: Hi, I have one Ovirt cluster and under this cluster all the guest machines (such as centos,ubuntu,debian etc) almost 160 RCU processes running i searched on google about the RCU (It's kernel process read-copy-upate) http://lwn.net/Articles/518953/ I want to know how i can modify this 160 processes to 10-20 or how i can to disable itis there any bed impact if i disable it Thanks, Punit Hi Punit, we need to do a bit of a research on this one. In order to make sure we keep track of it, do you mind opening an RFE (BZ) with all the relevant details, including the hardware that you're using, guests config, how busy the guests/host are, Which hypervisor is used and the versions. Thanks, Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Guest VM Running 160 RCU Processes
- Original Message - From: Punit Dambiwal hypu...@gmail.com To: Doron Fediuck dfedi...@redhat.com Sent: Wednesday, October 22, 2014 4:19:56 AM Subject: Re: [ovirt-users] Guest VM Running 160 RCU Processes Hi Darrell, I haven't open a BZ for the same...please let me know how i can open BZ ?? Doron@ yes i am referring the MaxNumOfVmCpus...as my host machine has 10 CPU with HT so totally 20cpus...thats' why i modify this setting with MaxNumOfVmCpus=20...as i can not assign 160 Vcpus as my host has 20 only Thanks, Punit Punit, did you set the HT flag in the cluster policy? See: http://www.ovirt.org/OVirt_Administration_Guide#Optimization_Settings_Explained You should set Count Threads As Cores or the engine will ignore your threads. On Wed, Oct 22, 2014 at 12:29 AM, Doron Fediuck dfedi...@redhat.com wrote: Hi Punit, are you referring to MaxNumOfVmCpus? If so I do not see how this will have an effect other than not allowing you to run VMs with too many CPUs. - Original Message - From: Punit Dambiwal hypu...@gmail.com To: Doron Fediuck dfedi...@redhat.com Sent: Tuesday, October 21, 2014 5:24:02 AM Subject: Re: [ovirt-users] Guest VM Running 160 RCU Processes Hi Doron, I solved the issue with lower down the max vcpu setting in the engine-config... Thanks, Punit On Mon, Oct 20, 2014 at 8:34 PM, Doron Fediuck dfedi...@redhat.com wrote: - Original Message - From: Punit Dambiwal hypu...@gmail.com To: users@ovirt.org, Dan Kenigsberg dan...@redhat.com, Itamar Heim ih...@redhat.com, aha...@redhat.com Sent: Monday, October 20, 2014 5:58:20 AM Subject: Re: [ovirt-users] Guest VM Running 160 RCU Processes Hi, Is there any body suggest me good way to handle it?? On Fri, Oct 17, 2014 at 3:15 PM, Punit Dambiwal hypu...@gmail.com wrote: Hi, I have one Ovirt cluster and under this cluster all the guest machines (such as centos,ubuntu,debian etc) almost 160 RCU processes running i searched on google about the RCU (It's kernel process read-copy-upate) http://lwn.net/Articles/518953/ I want to know how i can modify this 160 processes to 10-20 or how i can to disable itis there any bed impact if i disable it Thanks, Punit Hi Punit, we need to do a bit of a research on this one. In order to make sure we keep track of it, do you mind opening an RFE (BZ) with all the relevant details, including the hardware that you're using, guests config, how busy the guests/host are, Which hypervisor is used and the versions. Thanks, Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Guest VM Running 160 RCU Processes
Hi Doron, Yes.. HT Flag already enabled in my cluster policy... Thanks, Punit On Wed, Oct 22, 2014 at 4:04 PM, Doron Fediuck dfedi...@redhat.com wrote: - Original Message - From: Punit Dambiwal hypu...@gmail.com To: Doron Fediuck dfedi...@redhat.com Sent: Wednesday, October 22, 2014 4:19:56 AM Subject: Re: [ovirt-users] Guest VM Running 160 RCU Processes Hi Darrell, I haven't open a BZ for the same...please let me know how i can open BZ ?? Doron@ yes i am referring the MaxNumOfVmCpus...as my host machine has 10 CPU with HT so totally 20cpus...thats' why i modify this setting with MaxNumOfVmCpus=20...as i can not assign 160 Vcpus as my host has 20 only Thanks, Punit Punit, did you set the HT flag in the cluster policy? See: http://www.ovirt.org/OVirt_Administration_Guide#Optimization_Settings_Explained You should set Count Threads As Cores or the engine will ignore your threads. On Wed, Oct 22, 2014 at 12:29 AM, Doron Fediuck dfedi...@redhat.com wrote: Hi Punit, are you referring to MaxNumOfVmCpus? If so I do not see how this will have an effect other than not allowing you to run VMs with too many CPUs. - Original Message - From: Punit Dambiwal hypu...@gmail.com To: Doron Fediuck dfedi...@redhat.com Sent: Tuesday, October 21, 2014 5:24:02 AM Subject: Re: [ovirt-users] Guest VM Running 160 RCU Processes Hi Doron, I solved the issue with lower down the max vcpu setting in the engine-config... Thanks, Punit On Mon, Oct 20, 2014 at 8:34 PM, Doron Fediuck dfedi...@redhat.com wrote: - Original Message - From: Punit Dambiwal hypu...@gmail.com To: users@ovirt.org, Dan Kenigsberg dan...@redhat.com, Itamar Heim ih...@redhat.com, aha...@redhat.com Sent: Monday, October 20, 2014 5:58:20 AM Subject: Re: [ovirt-users] Guest VM Running 160 RCU Processes Hi, Is there any body suggest me good way to handle it?? On Fri, Oct 17, 2014 at 3:15 PM, Punit Dambiwal hypu...@gmail.com wrote: Hi, I have one Ovirt cluster and under this cluster all the guest machines (such as centos,ubuntu,debian etc) almost 160 RCU processes running i searched on google about the RCU (It's kernel process read-copy-upate) http://lwn.net/Articles/518953/ I want to know how i can modify this 160 processes to 10-20 or how i can to disable itis there any bed impact if i disable it Thanks, Punit Hi Punit, we need to do a bit of a research on this one. In order to make sure we keep track of it, do you mind opening an RFE (BZ) with all the relevant details, including the hardware that you're using, guests config, how busy the guests/host are, Which hypervisor is used and the versions. Thanks, Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Guest VM Running 160 RCU Processes
On Oct 21, 2014, at 18:13 , Darrell Budic bu...@onholyground.com wrote: Was poking at this a little to see if there was any tuning that could affect it and spotted some oddness with the processor counts on my VMs under Ovirt 3.4. They seem to think they only have the proper number I set in ovirt (as shown in /proc/cpu), but if I look at dmidecode, there’s 159 bogus processors listed. I’d expect maybe 16 from the -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 argument to qemu-kvm, but there are 0xa0 of them. Maybe this is a seabios or qemu-kvm issue causing all those extras? Anyway, the # of rcu* processes matches pretty well, so that’s likely where it’s coming from. At least they shouldn’t be causing a performance issue, given their purpose as non-blocking work threads, but it is odd to see. Punit, did you open a BZ I can add these details to? BTW, this appears to be corrected with some component of Ovirt 3.5 (probably qemu-rhev?). On VMs started after I upgraded my Engine (even on 3.4 vdsmds), I’m only seeing 16 “processors” in the bios, and thus only 16 of the various rcu* processes. Could have been a general Centos 6.5 update too, since I did those as well, so I can’t get any finer resolution on that issue (both engine and host nodes, lots of Centos 7 VMs). If there is a RFE for this, perhaps a configurable max # of CPU sockets for hot add could be added, or it could be limited to the max physical cpu count of the biggest host in the cluster? The max is 16. Roy, what can we change/not change after your latest changes? I suppose it's related to teh maximum values we send because of hotplug support Thanks, michal -Darrell On Oct 20, 2014, at 7:34 AM, Doron Fediuck dfedi...@redhat.com wrote: - Original Message - From: Punit Dambiwal hypu...@gmail.com To: users@ovirt.org, Dan Kenigsberg dan...@redhat.com, Itamar Heim ih...@redhat.com, aha...@redhat.com Sent: Monday, October 20, 2014 5:58:20 AM Subject: Re: [ovirt-users] Guest VM Running 160 RCU Processes Hi, Is there any body suggest me good way to handle it?? On Fri, Oct 17, 2014 at 3:15 PM, Punit Dambiwal hypu...@gmail.com wrote: Hi, I have one Ovirt cluster and under this cluster all the guest machines (such as centos,ubuntu,debian etc) almost 160 RCU processes running i searched on google about the RCU (It's kernel process read-copy-upate) http://lwn.net/Articles/518953/ I want to know how i can modify this 160 processes to 10-20 or how i can to disable itis there any bed impact if i disable it Thanks, Punit Hi Punit, we need to do a bit of a research on this one. In order to make sure we keep track of it, do you mind opening an RFE (BZ) with all the relevant details, including the hardware that you're using, guests config, how busy the guests/host are, Which hypervisor is used and the versions. Thanks, Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Guest VM Running 160 RCU Processes
On Oct 22, 2014, at 6:43 AM, Michal Skrivanek michal.skriva...@redhat.com wrote: On Oct 21, 2014, at 18:13 , Darrell Budic bu...@onholyground.com wrote: Was poking at this a little to see if there was any tuning that could affect it and spotted some oddness with the processor counts on my VMs under Ovirt 3.4. They seem to think they only have the proper number I set in ovirt (as shown in /proc/cpu), but if I look at dmidecode, there’s 159 bogus processors listed. I’d expect maybe 16 from the -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 argument to qemu-kvm, but there are 0xa0 of them. Maybe this is a seabios or qemu-kvm issue causing all those extras? Anyway, the # of rcu* processes matches pretty well, so that’s likely where it’s coming from. At least they shouldn’t be causing a performance issue, given their purpose as non-blocking work threads, but it is odd to see. Punit, did you open a BZ I can add these details to? BTW, this appears to be corrected with some component of Ovirt 3.5 (probably qemu-rhev?). On VMs started after I upgraded my Engine (even on 3.4 vdsmds), I’m only seeing 16 “processors” in the bios, and thus only 16 of the various rcu* processes. Could have been a general Centos 6.5 update too, since I did those as well, so I can’t get any finer resolution on that issue (both engine and host nodes, lots of Centos 7 VMs). If there is a RFE for this, perhaps a configurable max # of CPU sockets for hot add could be added, or it could be limited to the max physical cpu count of the biggest host in the cluster? The max is 16. Roy, what can we change/not change after your latest changes? I suppose it's related to teh maximum values we send because of hotplug support Thanks, michal Yes, definitely looks like a side effect of hot plug. The 160 rcu issues look like a bug somewhere else, 16 is more normal for the default. I had missed the engine setting for MaxNumOfVmCpus, I’ll probably set that for my smaller hosts just to keep it reasonable for the hardware it’s on. From: Punit Dambiwal hypu...@gmail.com To: Doron Fediuck dfedi...@redhat.com Sent: Wednesday, October 22, 2014 4:19:56 AM Subject: Re: [ovirt-users] Guest VM Running 160 RCU Processes Hi Darrell, I haven't open a BZ for the same...please let me know how i can open BZ ?? Punit, make sure your replies go to the list, this didn’t get to me. But you want to go to http://www.ovirt.org/Community and look at the “Find and File Bugs” section to raise a bug or file an RFE. -Darrell ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Guest VM Running 160 RCU Processes
Was poking at this a little to see if there was any tuning that could affect it and spotted some oddness with the processor counts on my VMs under Ovirt 3.4. They seem to think they only have the proper number I set in ovirt (as shown in /proc/cpu), but if I look at dmidecode, there’s 159 bogus processors listed. I’d expect maybe 16 from the -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 argument to qemu-kvm, but there are 0xa0 of them. Maybe this is a seabios or qemu-kvm issue causing all those extras? Anyway, the # of rcu* processes matches pretty well, so that’s likely where it’s coming from. At least they shouldn’t be causing a performance issue, given their purpose as non-blocking work threads, but it is odd to see. Punit, did you open a BZ I can add these details to? BTW, this appears to be corrected with some component of Ovirt 3.5 (probably qemu-rhev?). On VMs started after I upgraded my Engine (even on 3.4 vdsmds), I’m only seeing 16 “processors” in the bios, and thus only 16 of the various rcu* processes. Could have been a general Centos 6.5 update too, since I did those as well, so I can’t get any finer resolution on that issue (both engine and host nodes, lots of Centos 7 VMs). If there is a RFE for this, perhaps a configurable max # of CPU sockets for hot add could be added, or it could be limited to the max physical cpu count of the biggest host in the cluster? -Darrell On Oct 20, 2014, at 7:34 AM, Doron Fediuck dfedi...@redhat.com wrote: - Original Message - From: Punit Dambiwal hypu...@gmail.com To: users@ovirt.org, Dan Kenigsberg dan...@redhat.com, Itamar Heim ih...@redhat.com, aha...@redhat.com Sent: Monday, October 20, 2014 5:58:20 AM Subject: Re: [ovirt-users] Guest VM Running 160 RCU Processes Hi, Is there any body suggest me good way to handle it?? On Fri, Oct 17, 2014 at 3:15 PM, Punit Dambiwal hypu...@gmail.com wrote: Hi, I have one Ovirt cluster and under this cluster all the guest machines (such as centos,ubuntu,debian etc) almost 160 RCU processes running i searched on google about the RCU (It's kernel process read-copy-upate) http://lwn.net/Articles/518953/ I want to know how i can modify this 160 processes to 10-20 or how i can to disable itis there any bed impact if i disable it Thanks, Punit Hi Punit, we need to do a bit of a research on this one. In order to make sure we keep track of it, do you mind opening an RFE (BZ) with all the relevant details, including the hardware that you're using, guests config, how busy the guests/host are, Which hypervisor is used and the versions. Thanks, Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Guest VM Running 160 RCU Processes
- Original Message - From: Punit Dambiwal hypu...@gmail.com To: users@ovirt.org, Dan Kenigsberg dan...@redhat.com, Itamar Heim ih...@redhat.com, aha...@redhat.com Sent: Monday, October 20, 2014 5:58:20 AM Subject: Re: [ovirt-users] Guest VM Running 160 RCU Processes Hi, Is there any body suggest me good way to handle it?? On Fri, Oct 17, 2014 at 3:15 PM, Punit Dambiwal hypu...@gmail.com wrote: Hi, I have one Ovirt cluster and under this cluster all the guest machines (such as centos,ubuntu,debian etc) almost 160 RCU processes running i searched on google about the RCU (It's kernel process read-copy-upate) http://lwn.net/Articles/518953/ I want to know how i can modify this 160 processes to 10-20 or how i can to disable itis there any bed impact if i disable it Thanks, Punit Hi Punit, we need to do a bit of a research on this one. In order to make sure we keep track of it, do you mind opening an RFE (BZ) with all the relevant details, including the hardware that you're using, guests config, how busy the guests/host are, Which hypervisor is used and the versions. Thanks, Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Guest VM Running 160 RCU Processes
Hi, Is there any body suggest me good way to handle it?? On Fri, Oct 17, 2014 at 3:15 PM, Punit Dambiwal hypu...@gmail.com wrote: Hi, I have one Ovirt cluster and under this cluster all the guest machines (such as centos,ubuntu,debian etc) almost 160 RCU processes running i searched on google about the RCU (It's kernel process read-copy-upate) http://lwn.net/Articles/518953/ I want to know how i can modify this 160 processes to 10-20 or how i can to disable itis there any bed impact if i disable it Thanks, Punit ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Guest VM Running 160 RCU Processes
Hi, I have one Ovirt cluster and under this cluster all the guest machines (such as centos,ubuntu,debian etc) almost 160 RCU processes running i searched on google about the RCU (It's kernel process read-copy-upate) http://lwn.net/Articles/518953/ I want to know how i can modify this 160 processes to 10-20 or how i can to disable itis there any bed impact if i disable it Thanks, Punit ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
