Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-23 Thread Mark Greenall
Hi, So, on Thursday we had the worst scenario occur. All hosts in the 4 node cluster we've had these issues with went non responsive and starting looping through various states. Spread across these hosts are the 45 guests we have the 45 storage domains for. As we have a responsibility to the en

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-17 Thread Nir Soffer
On Tue, Jan 17, 2017 at 1:17 PM, Mark Greenall wrote: > Hi Nir, > > Thanks for your continuing efforts with this. It's really appreciated. > > re: number of storage domains, we'll plan a review of the underlying storage > configuration to try and optimise things better. Do you have a recommended

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-17 Thread Nir Soffer
On Mon, Jan 16, 2017 at 7:19 PM, Mark Greenall wrote: > Hi, > > To try and get a baseline here I've reverted most of the changes we've made > and am running the host with just the following iSCSI related configuration > settings. The tweaks had been made over time to try and alleviate several >

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-16 Thread Mark Greenall
Hi, To try and get a baseline here I've reverted most of the changes we've made and am running the host with just the following iSCSI related configuration settings. The tweaks had been made over time to try and alleviate several storage related problems, but it's possible that fixes in Ovirt (

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-15 Thread Nir Soffer
On Thu, Jan 12, 2017 at 12:02 PM, Mark Greenall wrote: > Firstly, thanks @Yaniv and thanks @Nir for your responses. > > @Yaniv, in answer to this: > >>> Why do you have 1 SD per VM? > > It's a combination of performance and ease of management. We ran some IO > tests with various configurations an

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-15 Thread Nir Soffer
On Fri, Jan 13, 2017 at 11:29 AM, Mark Greenall wrote: > Hi Nir, > > Thanks very much for your feedback. It's really useful information. I keep my > fingers crossed it leads to a solution for us. > > All the settings we currently have were to try and optimise the Equallogic > for Linux and Ovirt

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-13 Thread Mark Greenall
Just been catching up with all the threads and I saw mention of some iscsid.conf settings which reminded me we also changed some of those from default as per the Dell Optimizing SAN Environment for Linux Guide previously mentioned. Changed from default in /etc/iscsi/iscsid.conf node.session.ini

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-13 Thread Mark Greenall
Hi Nir, Thanks very much for your feedback. It's really useful information. I keep my fingers crossed it leads to a solution for us. All the settings we currently have were to try and optimise the Equallogic for Linux and Ovirt. The multipath config settings came from this Dell Forum thread re

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-13 Thread Nicolas Ecarnot
Hi Nir, Le 13/01/2017 à 00:10, Nir Soffer a écrit : On Thu, Jan 12, 2017 at 6:01 PM, Nicolas Ecarnot wrote: Hi, As we are using a very similar hardware and usage as Mark (Dell poweredge hosts, Dell Equallogic SAN, iSCSI, and tons of LUNs for all those VMs), I'm jumping into this thread. Can

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-12 Thread Gianluca Cecchi
On Fri, Jan 13, 2017 at 12:10 AM, Nir Soffer wrote: > On Thu, Jan 12, 2017 at 6:01 PM, Nicolas Ecarnot > wrote: > > Hi, > > > > As we are using a very similar hardware and usage as Mark (Dell poweredge > > hosts, Dell Equallogic SAN, iSCSI, and tons of LUNs for all those VMs), > I'm > > jumping

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-12 Thread Nir Soffer
On Thu, Jan 12, 2017 at 6:01 PM, Nicolas Ecarnot wrote: > Hi, > > As we are using a very similar hardware and usage as Mark (Dell poweredge > hosts, Dell Equallogic SAN, iSCSI, and tons of LUNs for all those VMs), I'm > jumping into this thread. Can you share your multipath.conf that works with D

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-12 Thread Nir Soffer
On Thu, Jan 12, 2017 at 12:02 PM, Mark Greenall wrote: > Firstly, thanks @Yaniv and thanks @Nir for your responses. > > @Yaniv, in answer to this: > >>> Why do you have 1 SD per VM? > > It's a combination of performance and ease of management. We ran some IO > tests with various configurations an

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-12 Thread Yaniv Kaul
On Thu, Jan 12, 2017 at 6:01 PM, Nicolas Ecarnot wrote: > Hi, > > As we are using a very similar hardware and usage as Mark (Dell poweredge > hosts, Dell Equallogic SAN, iSCSI, and tons of LUNs for all those VMs), I'm > jumping into this thread. > > Le 12/01/2017 à 16:29, Yaniv Kaul a écrit : > >

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-12 Thread Mark Greenall
Hi Yaniv, >> 1. There is no point in so many connections. >> 2. Certainly not the same portal - you really should have more. >> 3. Note that some go via bond1 - and some via 'default' interface. Is that >> intended? >> 4. Your multipath.conf is using rr_min_io - where it should use rr_min_io_rq

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-12 Thread Mark Greenall
>> I would say that each path has its own device (and therefore its own queue). >> So I'd argue that you may want to have (for example) 4 paths to each LUN or >> perhaps more (8?). For example, with 2 NICs, each connecting to two >> controllers, each controller having 2 NICs (so no SPOF and nice

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-12 Thread Nicolas Ecarnot
Hi, As we are using a very similar hardware and usage as Mark (Dell poweredge hosts, Dell Equallogic SAN, iSCSI, and tons of LUNs for all those VMs), I'm jumping into this thread. Le 12/01/2017 à 16:29, Yaniv Kaul a écrit : While it's a bit of a religious war on what is preferred with iSCSI

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-12 Thread Yaniv Kaul
On Thu, Jan 12, 2017 at 5:01 PM, Mark Greenall wrote: > Hi Yaniv, > > > > >> 1. There is no point in so many connections. > > >> 2. Certainly not the same portal - you really should have more. > > >> 3. Note that some go via bond1 - and some via 'default' interface. Is > that intended? > > >> 4.

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-12 Thread Yaniv Kaul
On Thu, Jan 12, 2017 at 12:02 PM, Mark Greenall wrote: > Firstly, thanks @Yaniv and thanks @Nir for your responses. > > @Yaniv, in answer to this: > > >> Why do you have 1 SD per VM? > > It's a combination of performance and ease of management. We ran some IO > tests with various configurations a

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-11 Thread Nir Soffer
On Wed, Jan 11, 2017 at 9:23 PM, Nir Soffer wrote: > On Wed, Jan 11, 2017 at 7:35 PM, Mark Greenall > wrote: >> Hi Ovirt Champions, >> >> >> >> I am pulling my hair out and in need of advice / help. >> >> >> >> Host server: Dell PowerEdge R815 (40 cores and 768GB memory) >> >> Stoage: Dell Equall

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-11 Thread Nir Soffer
On Wed, Jan 11, 2017 at 7:35 PM, Mark Greenall wrote: > Hi Ovirt Champions, > > > > I am pulling my hair out and in need of advice / help. > > > > Host server: Dell PowerEdge R815 (40 cores and 768GB memory) > > Stoage: Dell Equallogic (Firmware V8.1.4) > > OS: Centos 7.3 (although the same thing

Re: [ovirt-users] Ovirt host activation and lvm looping with high CPU load trying to mount iSCSI storage

2017-01-11 Thread Yaniv Kaul
On Wed, Jan 11, 2017 at 7:35 PM, Mark Greenall wrote: > Hi Ovirt Champions, > > > > I am pulling my hair out and in need of advice / help. > > > > Host server: Dell PowerEdge R815 (40 cores and 768GB memory) > > Stoage: Dell Equallogic (Firmware V8.1.4) > > OS: Centos 7.3 (although the same thing