[ovirt-users] Re: Tuning and testing GlusterFS performance

[William Dossett]

Ah. Ok… mine are the H710s and yes I had to do virtual drives at RAID 0.  I’ve 
got my first templates up and running now anyway, getting ready to demo this to 
mgmt. late this week or early next.  Hoping to get some budget for flash drives 
after that.

 

They got quotes in for renewing our VMware licensing last week… ½ a million!  
So I have a fairly interested audience 

 

Pretty sure with some cash I can get the performance we need using flash,  the 
other thing will be upgrades…  going to see how the upgrade from 4.2.4 to 4.2.5 
goes this week.  Classically this is where open source has failed me in the 
past, but this is feeling much more like a finished product than it used to.

 

Regards

Bill

 

 

From: Jayme  
Sent: Sunday, August 5, 2018 10:18 AM
To: William Dossett 
Cc: Darrell Budic ; users 
Subject: Re: [ovirt-users] Tuning and testing GlusterFS performance

 

I'm using h310s which are known to have crap queue depth, I'm using them 
because they are one of the only percs that allow you to do both raid and 
passtrhough jbod instead of having to jbod using individual raid 0s.  They 
should be fine but could bottleneck during an intensive brick rebuild in 
addition to regular volume activity 

 

On Sun, Aug 5, 2018, 1:06 PM William Dossett, mailto:william.doss...@gmail.com> > wrote:

I think Percs have queue depth of 31 if that’s of any help… fairly common with 
that level of controller.

 

From: Jayme mailto:jay...@gmail.com> > 
Sent: Sunday, August 5, 2018 9:50 AM
To: Darrell Budic mailto:bu...@onholyground.com> >
Cc: William Dossett mailto:william.doss...@gmail.com> >; users mailto:users@ovirt.org> >
Subject: Re: [ovirt-users] Tuning and testing GlusterFS performance

 

I would have to assume so because I have not manually modified any gluster 
volume settings after performing gdeploy via cockpit.  What would you recommend 
these values be set to and does the fact that I am running SSDs make any 
difference in this regard?  I've been a bit concerned about how a rebuild might 
affect performance as the raid controllers in these servers doesn't have a 
large queue depth 

 

On Sun, Aug 5, 2018, 12:07 PM Darrell Budic, mailto:bu...@onholyground.com> > wrote:

It set these by default?

 

cluster.shd-wait-qlength: 1

cluster.shd-max-threads: 8

 

In my experience, these are WAY too high and will degrade performance to the 
point of causing problems on decently used volumes during a heal. If these are 
being set by the HCI installer, I’d recommend changing them.

 

 

  _  

From: Jayme mailto:jay...@gmail.com> >
Subject: [ovirt-users] Re: Tuning and testing GlusterFS performance
Date: August 4, 2018 at 10:31:30 AM EDT
To: William Dossett
Cc: users

Yes the volume options can be changed on the fly post creation no problem.  
Good luck!

 

On Sat, Aug 4, 2018, 11:23 AM William Dossett, mailto:william.doss...@gmail.com> > wrote:

Hey, thanks!  Good catch!  Going to have to take a look at that, will be 
working on it this weekend.. hopefully we can do this post creation.

 

Thanks again

Bill

 

 

From: Jayme mailto:jay...@gmail.com> > 
Sent: Thursday, August 2, 2018 5:56 PM
To: William Dossett mailto:william.doss...@gmail.com> >
Cc: users mailto:users@ovirt.org> >
Subject: Re: [ovirt-users] Tuning and testing GlusterFS performance

 

Bill,

 

I thought I'd let you (and others know this) as it might save you some 
headaches.  I found that my performance problem was resolved by clicking 
"optimize for virt store" option in the volume settings of the hosted engine 
(for the data volume).  Doing this one change has increased my I/O performance 
by 10x alone.  I don't know why this would not be set or recommended by default 
but I'm glad I found it!

 

- James

 

On Thu, Aug 2, 2018 at 2:32 PM, William Dossett mailto:william.doss...@gmail.com> > wrote:

Yeah, I am just ramping up here, but this project is mostly on my own time and 
money, hence no SSDs for Gluster… I’ve already blown close to $500 of my own 
money on 10Gb ethernet cards and SFPs on ebay as my company frowns on us 
getting good deals for equipment on ebay and would rather go to their preferred 
supplier – where $500 wouldn’t even buy half a 10Gb CNA ☹  but I believe in 
this project and it feels like it is getting ready for showtime – if I can demo 
this in a few weeks and get some interest I’ll be asking them to reimburse me, 
that’s for sure!

 

Hopefully going to get some of the other work off my plate and work on this 
later this afternoon, will let you know any findings.

 

Regards

Bill

 

 

From: Jayme mailto:jay...@gmail.com> > 
Sent: Thursday, August 2, 2018 11:07 AM
To: William Dossett mailto:william.doss...@gmail.com> >
Cc: users mailto:users@ovirt.org> >
Subject: Re: [ovirt-users] Tuning and testing GlusterFS performance

 

Bill,

 

Appreciate the feedback and would be interested to hear some

[ovirt-users] Re: Tuning and testing GlusterFS performance

[William Dossett]

I think Percs have queue depth of 31 if that’s of any help… fairly common with 
that level of controller.

 

From: Jayme  
Sent: Sunday, August 5, 2018 9:50 AM
To: Darrell Budic 
Cc: William Dossett ; users 
Subject: Re: [ovirt-users] Tuning and testing GlusterFS performance

 

I would have to assume so because I have not manually modified any gluster 
volume settings after performing gdeploy via cockpit.  What would you recommend 
these values be set to and does the fact that I am running SSDs make any 
difference in this regard?  I've been a bit concerned about how a rebuild might 
affect performance as the raid controllers in these servers doesn't have a 
large queue depth 

 

On Sun, Aug 5, 2018, 12:07 PM Darrell Budic, mailto:bu...@onholyground.com> > wrote:

It set these by default?

 

cluster.shd-wait-qlength: 1

cluster.shd-max-threads: 8

 

In my experience, these are WAY too high and will degrade performance to the 
point of causing problems on decently used volumes during a heal. If these are 
being set by the HCI installer, I’d recommend changing them.

 





  _  

From: Jayme mailto:jay...@gmail.com> >
Subject: [ovirt-users] Re: Tuning and testing GlusterFS performance
Date: August 4, 2018 at 10:31:30 AM EDT
To: William Dossett
Cc: users

Yes the volume options can be changed on the fly post creation no problem.  
Good luck!

 

On Sat, Aug 4, 2018, 11:23 AM William Dossett, mailto:william.doss...@gmail.com> > wrote:

Hey, thanks!  Good catch!  Going to have to take a look at that, will be 
working on it this weekend.. hopefully we can do this post creation.

 

Thanks again

Bill

 

 

From: Jayme mailto:jay...@gmail.com> > 
Sent: Thursday, August 2, 2018 5:56 PM
To: William Dossett mailto:william.doss...@gmail.com> >
Cc: users mailto:users@ovirt.org> >
Subject: Re: [ovirt-users] Tuning and testing GlusterFS performance

 

Bill,

 

I thought I'd let you (and others know this) as it might save you some 
headaches.  I found that my performance problem was resolved by clicking 
"optimize for virt store" option in the volume settings of the hosted engine 
(for the data volume).  Doing this one change has increased my I/O performance 
by 10x alone.  I don't know why this would not be set or recommended by default 
but I'm glad I found it!

 

- James

 

On Thu, Aug 2, 2018 at 2:32 PM, William Dossett mailto:william.doss...@gmail.com> > wrote:

Yeah, I am just ramping up here, but this project is mostly on my own time and 
money, hence no SSDs for Gluster… I’ve already blown close to $500 of my own 
money on 10Gb ethernet cards and SFPs on ebay as my company frowns on us 
getting good deals for equipment on ebay and would rather go to their preferred 
supplier – where $500 wouldn’t even buy half a 10Gb CNA ☹  but I believe in 
this project and it feels like it is getting ready for showtime – if I can demo 
this in a few weeks and get some interest I’ll be asking them to reimburse me, 
that’s for sure!

 

Hopefully going to get some of the other work off my plate and work on this 
later this afternoon, will let you know any findings.

 

Regards

Bill

 

 

From: Jayme mailto:jay...@gmail.com> > 
Sent: Thursday, August 2, 2018 11:07 AM
To: William Dossett mailto:william.doss...@gmail.com> >
Cc: users mailto:users@ovirt.org> >
Subject: Re: [ovirt-users] Tuning and testing GlusterFS performance

 

Bill,

 

Appreciate the feedback and would be interested to hear some of your results.  
I'm a bit worried about what i'm seeing so far on a very stock 3 node HCI 
setup.  8mb/sec on that dd test mentioned in the original post from within a VM 
(which may be explained by bad testing methods or some other configuration 
considerations).. but what is more worrisome to me is that I tried another dd 
test to time creating a 32GB file, it was taking a long time so I exited the 
process and the VM basically locked up on me, I couldn't access it or the 
console and eventually had to do a hard shutdown of the VM to recover.  

 

I don't plan to host many VMs, probably around 15.  They aren't super demanding 
servers but some do read/write big directories such as working with github 
repos and large node_module folders, rsyncs of fairly large dirs etc.  I'm 
definitely going to have to do a lot more testing before I can be assured 
enough to put any important VMs on this cluster.

 

- James

 

On Thu, Aug 2, 2018 at 1:54 PM, William Dossett mailto:william.doss...@gmail.com> > wrote:

I usually look at IOPs using IOMeter… you usually want several workers running 
reads and writes in different threads at the same time.   You can run Dynamo on 
a Linux instance and then connect it to a window GUI running IOMeter to give 
you stats.  I was getting around 250 IOPs on JBOD sata 7200rpm drives which 
isn’t bad for cheap and cheerful sata drives.

 

As I said, I’ve worked with HCI in VMware now for a couple of years, i

[ovirt-users] Re: Tuning and testing GlusterFS performance

[William Dossett]

ID: 7065742b-c09d-410b-9e89-174ade4fc3f5

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 3 = 3

Transport-type: tcp

Bricks:

Brick1: MASKED:/gluster_bricks/vmstore/vmstore

Brick2: MASKED:/gluster_bricks/vmstore/vmstore

Brick3: MASKED:/gluster_bricks/vmstore/vmstore

Options Reconfigured:

cluster.granular-entry-heal: enable

performance.strict-o-direct: on

network.ping-timeout: 30

storage.owner-gid: 36

storage.owner-uid: 36

user.cifs: off

features.shard: on

cluster.shd-wait-qlength: 1

cluster.shd-max-threads: 8

cluster.locking-scheme: granular

cluster.data-self-heal-algorithm: full

cluster.server-quorum-type: server

cluster.quorum-type: auto

cluster.eager-lock: enable

network.remote-dio: off

performance.low-prio-threads: 32

performance.io-cache: off

performance.read-ahead: off

performance.quick-read: off

transport.address-family: inet

nfs.disable: on

performance.client-io-threads: off

 

Volume Name: vmstore2

Type: Replicate

Volume ID: 6f9a1c51-c0bc-46ad-b94a-fc2989a36e0c

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 3 = 3

Transport-type: tcp

Bricks:

Brick1: MASKED:/gluster_bricks/vmstore2/vmstore2

Brick2: MASKED:/gluster_bricks/vmstore2/vmstore2

Brick3: MASKED:/gluster_bricks/vmstore2/vmstore2

Options Reconfigured:

cluster.granular-entry-heal: enable

performance.strict-o-direct: on

network.ping-timeout: 30

storage.owner-gid: 36

storage.owner-uid: 36

user.cifs: off

features.shard: on

cluster.shd-wait-qlength: 1

cluster.shd-max-threads: 8

cluster.locking-scheme: granular

cluster.data-self-heal-algorithm: full

cluster.server-quorum-type: server

cluster.quorum-type: auto

cluster.eager-lock: enable

network.remote-dio: off

performance.low-prio-threads: 32

performance.io-cache: off

performance.read-ahead: off

performance.quick-read: off

transport.address-family: inet

nfs.disable: on

performance.client-io-threads: off

 

On Fri, Aug 3, 2018 at 6:53 AM, Sahina Bose mailto:sab...@redhat.com> > wrote:

 

 

On Fri, 3 Aug 2018 at 3:07 PM, Jayme mailto:jay...@gmail.com> > wrote:

Hello,

 

The option to optimize for virt store is tough to find (in my opinion) you have 
to go to volumes > volume name and then click the two dots to expand further 
options in the top right to see it.  No one would know to find it (or that it 
even exists) if they weren't specifically looking. 

 

I don't know enough about it but my assumption is that there are reasons why 
it's not set by default (as it might or should not need to apply to ever volume 
created), however my suggestion would be that it be included in the cockpit as 
a selectable option next to each volume you create with a hint to suggest that 
for best performance select it for any volume that is going to be a data volume 
for VMs

 

If you have installed via Cockpit, the options are set.

Can you provide the “gluster volume info “ output  after you optimised for virt?

 

 

 

. 

 

I simply installed using the latest node ISO / default cockpit deployment.

 

Hope this helps!

 

- Jayme

 

On Fri, Aug 3, 2018 at 5:15 AM, Sahina Bose mailto:sab...@redhat.com> > wrote:

 

 

On Fri, Aug 3, 2018 at 5:25 AM, Jayme mailto:jay...@gmail.com> > wrote:

Bill,

 

I thought I'd let you (and others know this) as it might save you some 
headaches.  I found that my performance problem was resolved by clicking 
"optimize for virt store" option in the volume settings of the hosted engine 
(for the data volume).  Doing this one change has increased my I/O performance 
by 10x alone.  I don't know why this would not be set or recommended by default 
but I'm glad I found it!

 

Thanks for the feedback, Could you log a bug to make it default by providing 
the user flow that you used.

 

Also, I would be interested to know how you prepared the gluster volume for use 
- if it was using the Cockpit deployment UI, the volume options would have been 
set by default.

 

 

- James

 

On Thu, Aug 2, 2018 at 2:32 PM, William Dossett mailto:william.doss...@gmail.com> > wrote:

Yeah, I am just ramping up here, but this project is mostly on my own time and 
money, hence no SSDs for Gluster… I’ve already blown close to $500 of my own 
money on 10Gb ethernet cards and SFPs on ebay as my company frowns on us 
getting good deals for equipment on ebay and would rather go to their preferred 
supplier – where $500 wouldn’t even buy half a 10Gb CNA ☹  but I believe in 
this project and it feels like it is getting ready for showtime – if I can demo 
this in a few weeks and get some interest I’ll be asking them to reimburse me, 
that’s for sure!

 

Hopefully going to get some of the other work off my plate and work on this 
later this afternoon, will let you know any findings.

 

Regards

Bill

 

 

From: Jayme mailto:jay...@gmail.com> > 
Sent: Thursday, August 2, 2018 11:07 AM
To: William Dossett mailto:william.doss...@gmail.com> >
Cc: users mailto:user

[ovirt-users] Re: Datacenter of several Hosts with Local Storage ? (share Logical Network)

[William Dossett]

Hi,

I'm not sure but this sounds like you want to do Hyper-Converged 
infrastructure, HCI... so a cluster of hosts with local storage...  There are a 
few of us working on testing this now.

If this is what you are looking for I would highly advise looking at 
https://www.ovirt.org/blog/2018/02/up-and-running-with-ovirt-4-2-and-gluster-storage/

Which explains how to get a 3 node cluster with local storage up and running.


-Original Message-
From: jeanbapti...@nfrance.com  
Sent: Friday, August 3, 2018 7:12 AM
To: users@ovirt.org
Subject: [ovirt-users] Datacenter of several Hosts with Local Storage ? (share 
Logical Network)

Hello,

My question is pretty simple :)
Is there a way to configure  Local(s) Storage Datacenter with more than 1 host ?

The goal is is to share some Datacenter simple parameters between several hosts 
(with Local storage) like Logical Networks ?
 If it is not possible what is the best way / workaound to do this ?
To compare, on Xenserver, it is possible to configure a "Pool" of some hosts 
which have Local Storage, but which share same Logical network configuration.. 
(I do not say XS is better^^, far from it !)

Thanks for all :)
Jean-Baptiste
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: 
https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6TO75UKJCYCGVQNSBHUX4QSOFCCSKCYP/
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BJYHPGSEDHFNRYS6YTUJQ7YFJFE7T46X/

[ovirt-users] Re: Tuning and testing GlusterFS performance

[William Dossett]

Hey, thanks!  Good catch!  Going to have to take a look at that, will be 
working on it this weekend.. hopefully we can do this post creation.

 

Thanks again

Bill

 

 

From: Jayme  
Sent: Thursday, August 2, 2018 5:56 PM
To: William Dossett 
Cc: users 
Subject: Re: [ovirt-users] Tuning and testing GlusterFS performance

 

Bill,

 

I thought I'd let you (and others know this) as it might save you some 
headaches.  I found that my performance problem was resolved by clicking 
"optimize for virt store" option in the volume settings of the hosted engine 
(for the data volume).  Doing this one change has increased my I/O performance 
by 10x alone.  I don't know why this would not be set or recommended by default 
but I'm glad I found it!

 

- James

 

On Thu, Aug 2, 2018 at 2:32 PM, William Dossett mailto:william.doss...@gmail.com> > wrote:

Yeah, I am just ramping up here, but this project is mostly on my own time and 
money, hence no SSDs for Gluster… I’ve already blown close to $500 of my own 
money on 10Gb ethernet cards and SFPs on ebay as my company frowns on us 
getting good deals for equipment on ebay and would rather go to their preferred 
supplier – where $500 wouldn’t even buy half a 10Gb CNA ☹  but I believe in 
this project and it feels like it is getting ready for showtime – if I can demo 
this in a few weeks and get some interest I’ll be asking them to reimburse me, 
that’s for sure!

 

Hopefully going to get some of the other work off my plate and work on this 
later this afternoon, will let you know any findings.

 

Regards

Bill

 

 

From: Jayme mailto:jay...@gmail.com> > 
Sent: Thursday, August 2, 2018 11:07 AM
To: William Dossett mailto:william.doss...@gmail.com> >
Cc: users mailto:users@ovirt.org> >
Subject: Re: [ovirt-users] Tuning and testing GlusterFS performance

 

Bill,

 

Appreciate the feedback and would be interested to hear some of your results.  
I'm a bit worried about what i'm seeing so far on a very stock 3 node HCI 
setup.  8mb/sec on that dd test mentioned in the original post from within a VM 
(which may be explained by bad testing methods or some other configuration 
considerations).. but what is more worrisome to me is that I tried another dd 
test to time creating a 32GB file, it was taking a long time so I exited the 
process and the VM basically locked up on me, I couldn't access it or the 
console and eventually had to do a hard shutdown of the VM to recover.  

 

I don't plan to host many VMs, probably around 15.  They aren't super demanding 
servers but some do read/write big directories such as working with github 
repos and large node_module folders, rsyncs of fairly large dirs etc.  I'm 
definitely going to have to do a lot more testing before I can be assured 
enough to put any important VMs on this cluster.

 

- James

 

On Thu, Aug 2, 2018 at 1:54 PM, William Dossett mailto:william.doss...@gmail.com> > wrote:

I usually look at IOPs using IOMeter… you usually want several workers running 
reads and writes in different threads at the same time.   You can run Dynamo on 
a Linux instance and then connect it to a window GUI running IOMeter to give 
you stats.  I was getting around 250 IOPs on JBOD sata 7200rpm drives which 
isn’t bad for cheap and cheerful sata drives.

 

As I said, I’ve worked with HCI in VMware now for a couple of years, intensely 
this last year when we had some defective Dell hardware and trying to diagnose 
the problem.  Since then the hardware has been completely replaced with all 
flash solution.   So when I got the all flash solution I used IOmeter on it and 
was only getting around 3000 IOPs on enterprise flash disks… not exactly 
stellar, but OK for one VM.  The trick there was the scale out.  There is a 
VMware Fling call HCI Bench.  Its very cool in that you spin up one VM and then 
it spawns 40 more VMs across the cluster.  I  could then use VSAN observer and 
it showed my hosts were actually doing 30K IOPs on average which is absolutely 
stellar performance.  

 

Anyway, moral of the story there was that your one VM may seem like its quick, 
but not what you would expect from flash…   but as you add more VMs in the 
cluster and they are all doing workloads, it scales out beautifully and the 
read/write speed does not slow down as you add more loads.  I’m hoping that’s 
what we are going to see with Gluster.

 

Also, you are using mb nomenclature below, is that Mb, or MB?  I am sort of 
assuming MB megabytes per second…  it does not seem very fast.  I’m probably 
not going to get to work more on my cluster today as I’ve got other projects 
that I need to get done on time, but I want to try and get some templates up 
and running and do some more testing either tomorrow or this weekend and see 
what I get in just basic writing MB/s and let you know.

 

Regards

Bill

 

 

From: Jayme mailto:jay...@gmail.com> > 
Sent: Thursday, August 2, 2018 8:12 AM
To: users mailto:us

[ovirt-users] Re: Tuning and testing GlusterFS performance

[William Dossett]

Yeah, I am just ramping up here, but this project is mostly on my own time and 
money, hence no SSDs for Gluster… I’ve already blown close to $500 of my own 
money on 10Gb ethernet cards and SFPs on ebay as my company frowns on us 
getting good deals for equipment on ebay and would rather go to their preferred 
supplier – where $500 wouldn’t even buy half a 10Gb CNA ☹  but I believe in 
this project and it feels like it is getting ready for showtime – if I can demo 
this in a few weeks and get some interest I’ll be asking them to reimburse me, 
that’s for sure!

 

Hopefully going to get some of the other work off my plate and work on this 
later this afternoon, will let you know any findings.

 

Regards

Bill

 

 

From: Jayme  
Sent: Thursday, August 2, 2018 11:07 AM
To: William Dossett 
Cc: users 
Subject: Re: [ovirt-users] Tuning and testing GlusterFS performance

 

Bill,

 

Appreciate the feedback and would be interested to hear some of your results.  
I'm a bit worried about what i'm seeing so far on a very stock 3 node HCI 
setup.  8mb/sec on that dd test mentioned in the original post from within a VM 
(which may be explained by bad testing methods or some other configuration 
considerations).. but what is more worrisome to me is that I tried another dd 
test to time creating a 32GB file, it was taking a long time so I exited the 
process and the VM basically locked up on me, I couldn't access it or the 
console and eventually had to do a hard shutdown of the VM to recover.  

 

I don't plan to host many VMs, probably around 15.  They aren't super demanding 
servers but some do read/write big directories such as working with github 
repos and large node_module folders, rsyncs of fairly large dirs etc.  I'm 
definitely going to have to do a lot more testing before I can be assured 
enough to put any important VMs on this cluster.

 

- James

 

On Thu, Aug 2, 2018 at 1:54 PM, William Dossett mailto:william.doss...@gmail.com> > wrote:

I usually look at IOPs using IOMeter… you usually want several workers running 
reads and writes in different threads at the same time.   You can run Dynamo on 
a Linux instance and then connect it to a window GUI running IOMeter to give 
you stats.  I was getting around 250 IOPs on JBOD sata 7200rpm drives which 
isn’t bad for cheap and cheerful sata drives.

 

As I said, I’ve worked with HCI in VMware now for a couple of years, intensely 
this last year when we had some defective Dell hardware and trying to diagnose 
the problem.  Since then the hardware has been completely replaced with all 
flash solution.   So when I got the all flash solution I used IOmeter on it and 
was only getting around 3000 IOPs on enterprise flash disks… not exactly 
stellar, but OK for one VM.  The trick there was the scale out.  There is a 
VMware Fling call HCI Bench.  Its very cool in that you spin up one VM and then 
it spawns 40 more VMs across the cluster.  I  could then use VSAN observer and 
it showed my hosts were actually doing 30K IOPs on average which is absolutely 
stellar performance.  

 

Anyway, moral of the story there was that your one VM may seem like its quick, 
but not what you would expect from flash…   but as you add more VMs in the 
cluster and they are all doing workloads, it scales out beautifully and the 
read/write speed does not slow down as you add more loads.  I’m hoping that’s 
what we are going to see with Gluster.

 

Also, you are using mb nomenclature below, is that Mb, or MB?  I am sort of 
assuming MB megabytes per second…  it does not seem very fast.  I’m probably 
not going to get to work more on my cluster today as I’ve got other projects 
that I need to get done on time, but I want to try and get some templates up 
and running and do some more testing either tomorrow or this weekend and see 
what I get in just basic writing MB/s and let you know.

 

Regards

Bill

 

 

From: Jayme mailto:jay...@gmail.com> > 
Sent: Thursday, August 2, 2018 8:12 AM
To: users mailto:users@ovirt.org> >
Subject: [ovirt-users] Tuning and testing GlusterFS performance

 

So I've finally completed my first HCI build using the below configuration:

 

3x

Dell PowerEdge R720

2x 2.9 GHz 8 Core E5-2690

256GB RAM

2x250gb SSD Raid 1 (boot/os)

2x2TB SSD jbod passthrough (used for gluster bricks)

1Gbe Nic for management 10Gbe nic for Gluster

 

Using Replica 3 with no arbiter. 

 

Installed the latest version of oVirt available at the time 4.2.5.  Created 
recommended volumes (with an additional data volume on second SSD). Not using 
VDO

 

First thing I did was setup glusterFS network on 10Gbe and set it to be used 
for glusterFS and migration traffic. 

 

I've setup a single test VM using Centos7 minimal on the default "x-large 
instance" profile. 

 

Within this VM if I do very basic write test using something like:

 

dd bs=1M count=256 if=/dev/zero of=test conv=fdatasync

 

I'm seeing quite slow speeds, only 8mb/sec.  

[ovirt-users] Re: Tuning and testing GlusterFS performance

[William Dossett]

I usually look at IOPs using IOMeter… you usually want several workers running 
reads and writes in different threads at the same time.   You can run Dynamo on 
a Linux instance and then connect it to a window GUI running IOMeter to give 
you stats.  I was getting around 250 IOPs on JBOD sata 7200rpm drives which 
isn’t bad for cheap and cheerful sata drives.

 

As I said, I’ve worked with HCI in VMware now for a couple of years, intensely 
this last year when we had some defective Dell hardware and trying to diagnose 
the problem.  Since then the hardware has been completely replaced with all 
flash solution.   So when I got the all flash solution I used IOmeter on it and 
was only getting around 3000 IOPs on enterprise flash disks… not exactly 
stellar, but OK for one VM.  The trick there was the scale out.  There is a 
VMware Fling call HCI Bench.  Its very cool in that you spin up one VM and then 
it spawns 40 more VMs across the cluster.  I  could then use VSAN observer and 
it showed my hosts were actually doing 30K IOPs on average which is absolutely 
stellar performance.  

 

Anyway, moral of the story there was that your one VM may seem like its quick, 
but not what you would expect from flash…   but as you add more VMs in the 
cluster and they are all doing workloads, it scales out beautifully and the 
read/write speed does not slow down as you add more loads.  I’m hoping that’s 
what we are going to see with Gluster.

 

Also, you are using mb nomenclature below, is that Mb, or MB?  I am sort of 
assuming MB megabytes per second…  it does not seem very fast.  I’m probably 
not going to get to work more on my cluster today as I’ve got other projects 
that I need to get done on time, but I want to try and get some templates up 
and running and do some more testing either tomorrow or this weekend and see 
what I get in just basic writing MB/s and let you know.

 

Regards

Bill

 

 

From: Jayme  
Sent: Thursday, August 2, 2018 8:12 AM
To: users 
Subject: [ovirt-users] Tuning and testing GlusterFS performance

 

So I've finally completed my first HCI build using the below configuration:

 

3x

Dell PowerEdge R720

2x 2.9 GHz 8 Core E5-2690

256GB RAM

2x250gb SSD Raid 1 (boot/os)

2x2TB SSD jbod passthrough (used for gluster bricks)

1Gbe Nic for management 10Gbe nic for Gluster

 

Using Replica 3 with no arbiter. 

 

Installed the latest version of oVirt available at the time 4.2.5.  Created 
recommended volumes (with an additional data volume on second SSD). Not using 
VDO

 

First thing I did was setup glusterFS network on 10Gbe and set it to be used 
for glusterFS and migration traffic. 

 

I've setup a single test VM using Centos7 minimal on the default "x-large 
instance" profile. 

 

Within this VM if I do very basic write test using something like:

 

dd bs=1M count=256 if=/dev/zero of=test conv=fdatasync

 

I'm seeing quite slow speeds, only 8mb/sec.  

 

If I do the same from one of the hosts gluster mounts i.e.

 

host1: /rhev/data-center/mnt/glusterSD/HOST:data 

 

I get about 30mb/sec (which still seems fairly low?)

 

Am I testing incorrectly here?  Is there anything I should be tuning on the 
Gluster volumes to increase performance with SSDs?  Where can I find out where 
the bottle neck is here, or is this expected performance of Gluster? 

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/YL4O7LQKXG6WTJGEW6GDNAL7RPULTZAW/

[ovirt-users] Re: uploading ISOs

[William Dossett]

Not convinced   chrome is easy enough, done it many times... I used to use 
firefox but there's a few web apps I use that it does not play well with...

One annoying thing about chrome is that if you clear your history and cookies, 
it blows away your certs as well.  Not sure if FF does that or not.

But thanks again, all up and running now!

-Original Message-
From: femi adegoke [mailto:ov...@fateknollogee.com] 
Sent: Wednesday, August 1, 2018 10:52 AM
To: users@ovirt.org
Subject: [ovirt-users] Re: uploading ISOs

Bill, the process of importing that cert is a lot easier in FireFox compared to 
Chrome!

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: 
https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PECJ4NLBXADPPC4KGIIDUE2QQ4R4MABW/
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/YSPJENNTIQ4SHQD5QEC3LYJBCXEQOYGI/

[ovirt-users] Re: uploading ISOs

[William Dossett]

Ok, why didn't I do that before ☹

Connection to ovirt-imageio-proxy service has failed. Make sure the service is 
installed, configured, and ovirt-engine certificate is registered as a valid CA 
in the browser.

Installed the cert and looks like it is ok now, thanks!

Bill

-Original Message-
From: femi adegoke [mailto:ov...@fateknollogee.com] 
Sent: Wednesday, August 1, 2018 8:18 AM
To: users@ovirt.org
Subject: [ovirt-users] Re: uploading ISOs

In the UI:
Storage, Disks, Upload, Start leads to am "Upload Image" popup.

When you hit the "Test Connection" button, is the test successful?
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: 
https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6L2V7X3HDZPYOGCH6NKIEOU3RPJGOYDC/
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/3WCJ5AIAJZV7ZTGF7NLHOZX4YXSCMYDF/

[ovirt-users] Re: uploading ISOs

[William Dossett]

Chrome, I have allowed pop-ups

-Original Message-
From: femi adegoke [mailto:ov...@fateknollogee.com] 
Sent: Wednesday, August 1, 2018 8:13 AM
To: users@ovirt.org
Subject: [ovirt-users] Re: uploading ISOs

What browser are you using to upload ISOs?
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: 
https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/A3EKIV2GQECAGUCO25INCLP6XNQJTQGS/
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WZPXLXOXC4CBXDOETVJEWL7KP6RKRQCZ/

[ovirt-users] Re: uploading ISOs

[William Dossett]

My destination domain is a vmstore on Gluster.  Its HCI environment.  The 
upload does not complete, it says Paused by System, resuming it, 15 seconds 
later goes back to Paused by System

 



 

I’m kind of at a loss how to proceed with my testing if I can’t get my ISOs 
uploaded….  I’m on 4.2.4.

 

Is it correct that this does not need to be an ISO domain?  It does not have to 
be NFS?

 

Thanks, appreciate any guidance.

 

Bill

 

 

From: Gianluca Cecchi [mailto:gianluca.cec...@gmail.com] 
Sent: Wednesday, August 1, 2018 7:02 AM
To: Eyal Shenitzky 
Cc: William Dossett ; Idan Shaby 
; Daniel Erez ; users 
Subject: Re: [ovirt-users] Re: uploading ISOs

 

On Wed, Aug 1, 2018 at 2:48 PM, Eyal Shenitzky mailto:eshen...@redhat.com> > wrote:

Adding Idan and Daniel.

 

 

 

Please keep in mind that there is currently a bug open where destination data 
domain is on block storage:

https://bugzilla.redhat.com/show_bug.cgi?id=1588400

 

The upload succeeds, but actually has failed and you are not able then to 
attach this iso to vm.

As I already wrote inside the bugzilla, in my opinion the scenario where one 
has block storage domains is the most important one for this feature so 
that he/she can completely avoid nfs if not needed for main storage

Hopefully fix will be in 4.2.6. I don't know how much actively worked on right 
now...

 

Gianluca

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/IUCY7L5JBTNN7DJNMXOBE37U4EKPQH5I/

[ovirt-users] Re: uploading ISOs

[William Dossett]

Thank you,  That was what I was using…. But I was told that an ISO domain is no 
longer necessary, is that correct?  I can upload to VMStore?  Or any Domain?

 

When I try to upload a Windows Server 2016 ISO, it starts to upload, then goes 
to Paused by System, if I resume, it goes back to Paused by system after about 
15 secs.

 

It is uploading to VMStore which has 498GB free of total 499 GB.

 

Thanks

Bill

 

 

From: Eyal Shenitzky [mailto:eshen...@redhat.com] 
Sent: Tuesday, July 31, 2018 10:24 PM
To: William Dossett 
Cc: users 
Subject: Re: [ovirt-users] uploading ISOs

 

Hi William,

 

You can now upload image (and ISO) using image-io.

You can find more information about it here - 
https://www.ovirt.org/develop/release-management/features/storage/image-upload/ 
.

 

On Wed, Aug 1, 2018 at 5:58 AM, William Dossett mailto:william.doss...@gmail.com> > wrote:

Hi,

 

Before I go to the work of making my iso’s available to the ovirt-engine so I 
can use the iso uploader cli…

 

I haven’t worked with ovirt for a while and the documents still refer to an ISO 
domain and engine-iso-uploader the former which I understand is not required 
now, just wondered if there was any easier way to upload through the web UI 
maybe?  I noticed an upload button under images…  could that be used for ISOs?

 

I remember that was something that put me off slightly working with Ovirt last 
time was the number of steps I had to take to get an ISO available to build a 
VM from, but if that’s still the way I will soldier on.

 

Thanks

 

Bill


___
Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> 
To unsubscribe send an email to users-le...@ovirt.org 
<mailto:users-le...@ovirt.org> 
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/G273A5HOLECYFM262BTA2Y7JSOH253DZ/





 

-- 

Regards,

Eyal Shenitzky

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/NMCJW5UXNF353DGIV73HU6FDTC7XSHAD/

[ovirt-users] uploading ISOs

[William Dossett]

Hi,

 

Before I go to the work of making my iso's available to the ovirt-engine so
I can use the iso uploader cli.

 

I haven't worked with ovirt for a while and the documents still refer to an
ISO domain and engine-iso-uploader the former which I understand is not
required now, just wondered if there was any easier way to upload through
the web UI maybe?  I noticed an upload button under images.  could that be
used for ISOs?

 

I remember that was something that put me off slightly working with Ovirt
last time was the number of steps I had to take to get an ISO available to
build a VM from, but if that's still the way I will soldier on.

 

Thanks

 

Bill

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/G273A5HOLECYFM262BTA2Y7JSOH253DZ/

[ovirt-users] Re: adding hosts problems with GlusterFS

[William Dossett]

Blacklisting the disks in multipath.conf did the trick, all good now.

 

When adding a new host, the process should run that blacklist all disks shell 
script like the cockpit deploy does… if I may be so bold to suggest 

 

From: Jayme [mailto:jay...@gmail.com] 
Sent: Tuesday, July 31, 2018 9:48 AM
To: Gobinda Das 
Cc: William Dossett ; Bill Dossett 
; users 
Subject: Re: [ovirt-users] Re: adding hosts problems with GlusterFS

 

Try doing a multipath -ll  and if that displays anything try flushing with 
multipath -F

 

On Tue, Jul 31, 2018 at 10:55 AM, Gobinda Das mailto:go...@redhat.com> > wrote:

Is the brick creation failing from UI? Can you run "gluster volume status" and 
check brick created or not?

If bricks creation failing then Looks like multipath is causing issue.I think 
you need to clear multipath entries and try.

 

On Tue, Jul 31, 2018 at 7:01 PM, William Dossett mailto:william.doss...@gmail.com> > wrote:

Yeah, for sure, I think I made it over my last one yesterday…. Going to start 
building out my storage today.  I had to get down into the OS and work with the 
logical volume manager commands a bit to clean up some of my mess and also 
figured out how to blacklist the disks on my 4th and 5th host from multipath…. 
So that’s encouraging…  if you need a hand feel free to ping me.

 

From: Jayme [mailto:jay...@gmail.com <mailto:jay...@gmail.com> ] 
Sent: Tuesday, July 31, 2018 5:11 AM
To: Bill Dossett mailto:bill.doss...@pb.com> >
Cc: users mailto:users@ovirt.org> >
Subject: [ovirt-users] Re: adding hosts problems with GlusterFS

 

I think we are in a race to see who can get a new HCI setup built faster.  
There have been a few hurtles along the way :)

 

On Tue, Jul 31, 2018, 6:53 AM Bill Dossett, mailto:bill.doss...@pb.com> > wrote:

Hmm, resounding silence….  Have attached a screen shot of the host storage 
devices on the host I added after the initial deployment.  So at the bottom you 
can see that sdb – sdg are all locked and FS type is multipath_member  those 
all equate to the UID type NAME at the top.  The very top row, I created a 
brick, so it is also now locked, however when I go to brick view there is no 
brick.  I also can’t seem to delete any brick one it is created either.  On the 
first 3 hosts, sdb – sdg are all unlocked and I can create bricks…  going to 
remove a host and re-add it to see if there is anything I am missing, but I 
don’t have any other ideas at this point.

 



 

 


Regards, Bill

 

Bill Dossett

DevOps Ninja

 <https://aws.amazon.com/certification/certified-devops-engineer-professional/> 
AWS-DevOps-Pro, 
<https://aws.amazon.com/certification/certified-solutions-architect-associate/> 
 AWS-SA-A   <http://mylearn.vmware.com/mgrReg/plan.cfm?plan=30483=www_cert> 
VCAP-DCA,  <http://mylearn.vmware.com/mgrReg/plan.cfm?plan=30484=www_cert> 
VCAP-DCD, MCSA Win Server

 


IT – Software Innovation Solutions

 

M +1 303-725 8163

bill.doss...@pb.com <mailto:bill.doss...@pb.com> 

 <http://pitneybowes.com> pitneybowes.com


Pitney Bowes

4750 Walnut Street 
<https://maps.google.com/?q=4750+Walnut+Street+Boulder+CO,+80301=gmail=g>
 

Boulder CO, 80301 
<https://maps.google.com/?q=4750+Walnut+Street+Boulder+CO,+80301=gmail=g>
 



 <http://teamsite.inside.pb.com/sites/TCPPM/GES/SitePages/Home.aspx> Learn more 
about Software Innovation Solutions

 

In Engineering, Software Support or Professional Services?

Raise a ticket via Remedy Anywhere [ 
<http://teamsite.inside.pb.com/sites/TCPPM/GES/Lists/Create%20Incident/newIncident.aspx>
 HERE] takes less than a minute

 

CloudForms User Guide available [ 
<https://teamsite.inside.pb.com/sites/TCPPM/GES/SiteAssets/TC%20Knowledge%20Base/Cloudforms%20User%20Guide/Cloudforms%20-%20User%20Manual%20PB%20v3.pdf>
 HERE]

 

 

 

___
Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> 
To unsubscribe send an email to users-le...@ovirt.org 
<mailto:users-le...@ovirt.org> 
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AR3D4HLVTAQPEU774BP5LQYRHPP4234X/


___
Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> 
To unsubscribe send an email to users-le...@ovirt.org 
<mailto:users-le...@ovirt.org> 
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/

List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WOGYCVC5YLEYHH6HDHABIH2P5EQ7BQZO/





 

-- 

Thanks,

Gobinda

 

___
Users mailing list -- users@ovirt.org
To unsubscribe send an em

[ovirt-users] Re: Hosted Engine deploy failed on new HCI build

[William Dossett]

Can you connect to it with VNC?   That was a new one on me … since last working 
with this.  Actually this is the first time I did hosted engine deploy as I 
used to use a VM on my vmware infra to do it so had no idea I could connect to 
the hosted engine console until Simone helped me.

 

Hopefully you can connect with VNC … I was able to find the networking issue 
very quickly that way, a LOT faster than starting over.  I wiped and started 
over like 5 times, but learned something each time.

 

I’m around most of today on and off and trying to fix some code for another 
project but  happy to have a look with you if you want to ping me.

 

Regards

Bill

 

 

From: Simone Tiraboschi [mailto:stira...@redhat.com] 
Sent: Tuesday, July 31, 2018 6:35 AM
To: William Dossett 
Cc: Jayme ; users 
Subject: Re: [ovirt-users] Re: Hosted Engine deploy failed on new HCI build

 

 

On Tue, Jul 31, 2018 at 2:51 AM William Dossett mailto:william.doss...@gmail.com> > wrote:

That happened to me twice… the second time I figure it out and it was 
networking.  

 

I am not familiar with the hosted-engine ---console…

 

The issue with the console device will be fixed for new deployment as for 
https://bugzilla.redhat.com/show_bug.cgi?id=1561964

 

If you deployed in the past, you have to edit the definition of the engine VM 
on the engine enabling the serial console device and reboot it.

in the mean time you can also use VNC as detailed below.

 

 

The person that helped me said to do the following:

 

Run on your first host
   hosted-engine --add-console-password
to set a temporary VNC password and then connect to it over VNC with something 
like
   remote-viewer vnc://:

 

Which got me in and allowed me to fix the networking once I saw what was wrong… 
 can you get to the console like that?

 

Regards

Bill

 

 

 

From: Jayme [mailto:jay...@gmail.com <mailto:jay...@gmail.com> ] 
Sent: Monday, July 30, 2018 3:38 PM
To: users mailto:users@ovirt.org> >
Subject: [ovirt-users] Re: Hosted Engine deploy failed on new HCI build

 

I haven't had much luck with this yet I completely wiped the three hosts and 
did the entire install over again from the ground up only this time I used dhcp 
instead of static IP for the hostedengine deployment and ended up failing again 
in the exact step as before, waiting for the VM to come back but never does. 

 

I still feel like it could be network related in some way just not sure how.  
Any ideas? 

 

On Mon, Jul 30, 2018, 2:25 PM Jayme, mailto:jay...@gmail.com> > wrote:

Latest version of oVirt node 4.2 installed on three hosts.  I completed 
successfully the cockpit gdeploy process to deploy HCI.  All of that went well 
with no errors.  I then proceeded to the hosted engine deployment step which 
eventually failed (log attached). 

 

This is the current status:

 

--== Host 1 status ==--

 

conf_on_shared_storage : True

Status up-to-date  : True

Hostname   : MASKED

Host ID: 1

Engine status  : {"reason": "failed liveliness check", 
"health": "bad", "vm": "up", "detail": "Up"}

 

This indicates that the engine VM is up at libvirt eyes but the engine could 
not be reached over the network.

I'd suggest to open a console to the engine VM (also the VNC one) and check 
it's network configuration.

If you are using DHCP, do you have a working reservation for that?

 

Score  : 3400

stopped: False

Local maintenance  : False

crc32  : 3fa48e03

local_conf_timestamp   : 8468

Host timestamp : 8468

Extra metadata (valid at timestamp):

metadata_parse_version=1

metadata_feature_version=1

timestamp=8468 (Mon Jul 30 14:21:09 2018)

host-id=1

score=3400

vm_conf_refresh_time=8468 (Mon Jul 30 14:21:09 2018)

conf_on_shared_storage=True

maintenance=False

state=EngineStarting

stopped=False

 

If I do hosted-engine --console I get:

 

The engine VM is running on this host

Connected to domain HostedEngine

Escape character is ^]

error: internal error: cannot find character device 

 

does anyone know why it may have failed or what I could do to recover from 
this?  I'm thinking it could have potentially failed due to some problem with 
network config.  If I could get a console in to the engine VM I might be able 
to fix it but that serial error above is preventing me from reaching the vm 
console to diagnose further. 

 

Log of deploy attached:

 

Thanks!

 

 

___
Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> 
To unsubscribe send an email to users-le...@ovirt.org 
<mailto:users-le...@ovirt.org> 
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt C

[ovirt-users] Re: adding hosts problems with GlusterFS

[William Dossett]

Yeah, for sure, I think I made it over my last one yesterday…. Going to start 
building out my storage today.  I had to get down into the OS and work with the 
logical volume manager commands a bit to clean up some of my mess and also 
figured out how to blacklist the disks on my 4th and 5th host from multipath…. 
So that’s encouraging…  if you need a hand feel free to ping me.

 

From: Jayme [mailto:jay...@gmail.com] 
Sent: Tuesday, July 31, 2018 5:11 AM
To: Bill Dossett 
Cc: users 
Subject: [ovirt-users] Re: adding hosts problems with GlusterFS

 

I think we are in a race to see who can get a new HCI setup built faster.  
There have been a few hurtles along the way :)

 

On Tue, Jul 31, 2018, 6:53 AM Bill Dossett, mailto:bill.doss...@pb.com> > wrote:

Hmm, resounding silence….  Have attached a screen shot of the host storage 
devices on the host I added after the initial deployment.  So at the bottom you 
can see that sdb – sdg are all locked and FS type is multipath_member  those 
all equate to the UID type NAME at the top.  The very top row, I created a 
brick, so it is also now locked, however when I go to brick view there is no 
brick.  I also can’t seem to delete any brick one it is created either.  On the 
first 3 hosts, sdb – sdg are all unlocked and I can create bricks…  going to 
remove a host and re-add it to see if there is anything I am missing, but I 
don’t have any other ideas at this point.

 



 

 


Regards, Bill

 

Bill Dossett

DevOps Ninja

  
AWS-DevOps-Pro, 
 
 AWS-SA-A    
VCAP-DCA,   
VCAP-DCD, MCSA Win Server

 


IT – Software Innovation Solutions

 

M +1 303-725 8163

bill.doss...@pb.com  

  pitneybowes.com


Pitney Bowes

4750 Walnut Street

Boulder CO, 80301





  Learn more 
about Software Innovation Solutions

 

In Engineering, Software Support or Professional Services?

Raise a ticket via Remedy Anywhere [ 

 HERE] takes less than a minute

 

CloudForms User Guide available [ 

 HERE]

 

 

 

___
Users mailing list -- users@ovirt.org  
To unsubscribe send an email to users-le...@ovirt.org 
 
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AR3D4HLVTAQPEU774BP5LQYRHPP4234X/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WOGYCVC5YLEYHH6HDHABIH2P5EQ7BQZO/

[ovirt-users] Re: Hosted Engine deploy failed on new HCI build

[William Dossett]

That happened to me twice… the second time I figure it out and it was 
networking.  

 

I am not familiar with the hosted-engine ---console…

 

The person that helped me said to do the following:

 

Run on your first host
   hosted-engine --add-console-password
to set a temporary VNC password and then connect to it over VNC with something 
like
   remote-viewer vnc://:

 

Which got me in and allowed me to fix the networking once I saw what was wrong… 
 can you get to the console like that?

 

Regards

Bill

 

 

 

From: Jayme [mailto:jay...@gmail.com] 
Sent: Monday, July 30, 2018 3:38 PM
To: users 
Subject: [ovirt-users] Re: Hosted Engine deploy failed on new HCI build

 

I haven't had much luck with this yet I completely wiped the three hosts and 
did the entire install over again from the ground up only this time I used dhcp 
instead of static IP for the hostedengine deployment and ended up failing again 
in the exact step as before, waiting for the VM to come back but never does. 

 

I still feel like it could be network related in some way just not sure how.  
Any ideas? 

 

On Mon, Jul 30, 2018, 2:25 PM Jayme, mailto:jay...@gmail.com> > wrote:

Latest version of oVirt node 4.2 installed on three hosts.  I completed 
successfully the cockpit gdeploy process to deploy HCI.  All of that went well 
with no errors.  I then proceeded to the hosted engine deployment step which 
eventually failed (log attached). 

 

This is the current status:

 

--== Host 1 status ==--

 

conf_on_shared_storage : True

Status up-to-date  : True

Hostname   : MASKED

Host ID: 1

Engine status  : {"reason": "failed liveliness check", 
"health": "bad", "vm": "up", "detail": "Up"}

Score  : 3400

stopped: False

Local maintenance  : False

crc32  : 3fa48e03

local_conf_timestamp   : 8468

Host timestamp : 8468

Extra metadata (valid at timestamp):

metadata_parse_version=1

metadata_feature_version=1

timestamp=8468 (Mon Jul 30 14:21:09 2018)

host-id=1

score=3400

vm_conf_refresh_time=8468 (Mon Jul 30 14:21:09 2018)

conf_on_shared_storage=True

maintenance=False

state=EngineStarting

stopped=False

 

If I do hosted-engine --console I get:

 

The engine VM is running on this host

Connected to domain HostedEngine

Escape character is ^]

error: internal error: cannot find character device 

 

does anyone know why it may have failed or what I could do to recover from 
this?  I'm thinking it could have potentially failed due to some problem with 
network config.  If I could get a console in to the engine VM I might be able 
to fix it but that serial error above is preventing me from reaching the vm 
console to diagnose further. 

 

Log of deploy attached:

 

Thanks!

 

 

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WQDUJCZ46GMWDKKYIQZXGVKIORLT3VBN/

[ovirt-users] Re: adding hosts problems with GlusterFS

[william . dossett]

I found the problem...

with the cockpit deploy, all disks are blacklisted from multipath by a script 
called blacklist_all_disks.sh

This does not happen when a host is added manually.   To work around the 
problem edit /etc/multipath.conf  add the second line to the file below... this 
makes sure that vdsm will not ever mondify this file again.

 VDSM REVISION 1.5
# VDSM PRIVATE

then add the following to the end of the file


blacklist {
devnode "*"
}

this blacklists all disk from multipath.

Had to dig fairly deep for this, but now its working.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TFJKSIJ7SMZF2NBUZJ5GUDPHKERDZLGV/

[ovirt-users] Re: adding hosts problems with GlusterFS

[william . dossett]

unfortunately I can't access that bug with any of my redhat accounts.  I tried 
to make a bugzilla account and it says it is restricted access to that bug, 
internal only.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FGUSCJVWCTXWHJJJYEUSTQF3JCHBPSB3/

[ovirt-users] Re: adding hosts problems with GlusterFS

[william . dossett]

I did it from the UI.  I've just removed and re-added the host and I can't see 
any thing different I could do while adding it.  all the disks are locked.  

the brick is mounted in /gluster_bricks, but does not show in the bricks tab... 
 actually I think it may have been in the new volume view though.  I didn't 
actually want that volume though now that I realize I don't need an ISO domain 
anymore.  I am trying to clean this up now and try again.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZBQ2ILAEWHWLGCEBO3YJJCMAJQGQDLY3/

[ovirt-users] Re: up and running with ovirt 4.2 and gluster

[William Dossett]

Same here… I am using Dell R710s with 8 disks,  2 x 1TB in RAID 0 and then 6 
virtual disks, one per physical disks.  I think you have to do it that way for 
the dell perc controller to present the disk to the OS but I haven’t actually 
tried not doing that so not sure.. anyway this works.

 

This has been very informative.  

 

James has only 3 nodes, but I have 5…  I have another posting about what 
happens when I added the two additional nodes as all the disks seem to be 
locked and multipath and I can’t create bricks on them.

 

But, once I get pas that problem, if I have 5 nodes, and I am going to create 
replica datasttores with 3 disks, do I just 3 nodes for each datastore?  Is 
there any best practice for creating datastores on any 3 nodes…

 

Perhaps an example:

 

Datastore 1 -  Node1, Node2, Node3 /dev/sdb

Datastore 2 – Node2, Node3 /dev/sdc Node4 /dev/sdb

Datastore 3 -  Node3, /dev/sdd, Node4 /dev/sdc Node5 /dev/sda

 

Does that make sense?  Or is there a better way?

 

Thanks to everyone for helping me on this, much appreciated.

 

Bill

 

 

From: Jayme [mailto:jay...@gmail.com] 
Sent: Monday, July 30, 2018 8:12 AM
To: femi adegoke 
Cc: users 
Subject: [ovirt-users] Re: up and running with ovirt 4.2 and gluster

 

Hello,

 

In my case I'm referring to hosts with internal disks. 

 

Example of my hardware config:

 

host0:

2x250GB SSDs in RAID 1 mirror for OS/boot disk

2x2TB SSDs in JBOD (passthrough) represented as separate disks in the OS ex. 
/dev/sda and /dev/sdb

 

host1 and host2 have identical configurations.

 

- Jayme

 

On Mon, Jul 30, 2018 at 11:08 AM, femi adegoke mailto:ov...@fateknollogee.com> > wrote:

@jayme
@william.dossett

When you say "JBOD", are these hosts with xx number of disks or hosts with a 
physically attached JBOD?
___
Users mailing list -- users@ovirt.org  
To unsubscribe send an email to users-le...@ovirt.org 
 
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MWGC54O3OYM4AADG6JEIT2IHOQMEBH4F/

 

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/OGMWPUQNPPSUXPAPHHSVGY7QXDZXWKGQ/

[ovirt-users] Re: up and running with ovirt 4.2 and gluster

[william . dossett]

Hi James... I'm in sort of the same position... I have 6 JBOD drives in each 
node.  I am unsure if I am going to wind up with 6 data stores.  I have 5 nodes 
and have other problems related to when I add those nodes further complicating 
the gluster issue.  Hopefully someone can help clear this up as its the last 
hurdle I face before starting to actually put work loads on this and connecting 
it into our ManageIQ environment.  
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/VX5CRCDEF4XJBT5LP3VQ6QULSOYX3UTR/

[ovirt-users] Re: up and running with ovirt 4.2 and gluster

[william . dossett]

I would be happy to work on documentation... not quite sure how I start doing 
that though.  The previous post and others that detail the only actualy 
datastores needed are data and engine was helpful.  Having an additional 
datastore for isos that would perhaps not need to be (frequently) backed up 
also makes sense if you had a great deal of them.  I am still  not that 
familiar with the documentation - I used the up and running with ovirt 4.2 blog 
to get this setup.  Thanks to all that replied.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ESALHROBQHCOPSJQJ3KWKWY4A5ZXLOOT/

[ovirt-users] Re: up and running with ovirt 4.2 and gluster

[william . dossett]

Thank you, that clears things up!
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/7BJIJEYSMFSAGOAVH6A2YL2U47UVWU44/

[ovirt-users] adding hosts problems with GlusterFS

[william . dossett]

Hi,  I setup my 3 host HCI cluster and things are going pretty good.  I have a 
few issues though... 

one of them is that I added two more hosts to the three node cluster to make it 
a 5 nodes.   This seemed to work fine until I went to create bricks.  I have 6 
x 2 TB disks for use as JBOD in each node.

After I added the nodes, I went to the host and storage devices.   This looks 
significantly different that the view on my first 3 hosts.  On the two new 
hosts all my drives sdb - sdg have locks next to  them and file system type is 
multipath__member.   Further up the page then I see a UID under the name and 
description is PERC H710P dm-multipath - no lock next to it.

I created a brick on this and it created the lvmpv file system, however when I 
go to bricks view, there are no bricks.

So,  did I make a mistake while adding the hosts?  is there some way to prevent 
this dm-multipath configuration?

Thanks
Bill
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/CMU774UMB33KXG7W3UWJEXBQJMUBZI5N/

[ovirt-users] Re: trunked ports

[william . dossett]

just got back to this...  the method I usee worked well.

I used a 1GB copper port in my mgmt VLAN to set up oVirt with Gluster

Got it all up and running  then used 1 x 10 GB port to create my storage network

I think created to logical networks, one tagged with my static VLAN and the 
other tagged with my dhcp VLAN, dragged them both onto the second 10Gb port 
which is connected to cisco FEX trunk port - worked perfectly and I can deploy 
to either VLAN now.  I guess I will keep the 1Gb for management... I think I 
can transfer it to the 10Gb VLANs most likely, but didn't want to lose my ovirt 
engine as this setup seems to be working well for me at least on the network 
side.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/5X7BD5CMQF5VPLE4JU64XM2J4R64OA54/

[ovirt-users] Re: ovirt and gluster network config

[william . dossett]

follow up to this... i've gotten past the initial bump... obviously need a 
completely unused volume to gdeploy gluster to.  may be too early to comment, 
but I am in the middle of the lengthy play to deploy the hosted engine - and it 
looks like it is working.  I oversized some of the volumes and then had to go 
vgremove and pvremove on the nodes to clean up and start the deployment over 
which was a pain, but once I got past that this is starting to feel a lot like 
deployment of VXrail in that it is pretty much totally automated.  nice.  
looking forward to working with this!  Hopefully will be be giving a 
presentation to our tech leaders next week.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FQ5AKPKCZI3IEQ2TYKXOA2PLRZE2VF6D/

[ovirt-users] ovirt and gluster network config

[william . dossett]

Hi,

I have my 5 ovirt nodes up and running and starting to move forward with actual 
ovirt and gluster setup.  

I am using gdeploy for deployment.

the play book fails saying /dev/sdb is blacklisted by filter?  I am following 
the blog up and running on the ovirt site.

It mentioned adding 

# disable multipath
[script2]
action=execute
file=/usr/share/ansible/gdeploy/scripts/disable-multipath.sh

to get it to work, but it does not work.  The script 3 section after this does 
have the 

file=/usr/share/gdeploy/scripts/blacklist_all_disks.sh

in it.

What can I do to get past this? 

Also not sure if I should be modifying the script/play as I don't really want 
an arbitrator host - if I have 5 real hosts all with the same storage in them?  
(10TB in each)  and it only account for 3 hosts, can I add the other two later 
or should I try to modify the playbook to include these initially?  I am sure I 
will be setting this up several times before I get it right, but at the moment 
I can't get past this error.  Any help would really be appreciated!

Thanks
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/RD3SNZECTRJK33VIFV7SJUB4RQZPKNJH/

[ovirt-users] Re: ovirt and gluster disk config

[William Dossett]

Hi Jayme, thanks for the info… yes I’ve been googling and googling… I run a few 
VMware VSANs at the moment so HCI is my specialty … even one in my basement   
going to upgrade that to all flash soon as the 2TB disks are getting cheaper!  

 

I tried doing this a couple of years back as my post says but it was pretty 
much undoable then.  I guess I am going to go h/w raid to start and see how 
that is.  I haven’t got the money to put SSD in the 720s, but I do have 10Gb 
networking so hoping that will help on speed…  anyway, it’s a PoC so I am sure 
I will building and rebuilding a few times and trying to remember what I know 
of GlusterFS.

 

Regards

Bill

 

 

 

From: Jayme [mailto:jay...@gmail.com] 
Sent: Wednesday, May 23, 2018 6:25 AM
To: william.doss...@gmail.com
Cc: users 
Subject: Re: [ovirt-users] ovirt and gluster disk config

 

Unfortunately there is not a lot of info out there about ideal hardware configs 
for hyperconverged oVirt (at least not that I have found to date, and believe 
me I've looked).  If you have enough drives and especially if they are spinning 
disks as far as I know your best option would likely be to do a RAID-10 config 
to make up a single glusterfs brick on each physical server.  Raid-10 will add 
some performance benefits and also give you some options to replace failed 
disks without bringing the brick offline.  The downfall is that you waste a ton 
of usable space with Raid + gluster because each offer redundancy.  

 

I'm about to build a 3 node hyperconverted oVirt setup on Dell R720s myself, 
but I'm planning on using two 2Tb SSDs in JBOD configuration per host w/ a 
gluster replica 3.  I've gone back and forth on using JBOD or Raid config with 
SSDs and never did get a clear answer (online or asking on this mailing list).  
SSDs are too expensive right now to lose half usable space to raid then another 
half or more to gluster replication. 

 

On Tue, May 22, 2018 at 7:17 PM,  > wrote:

Hey,  been a few years since I looked at ovirt.   3 years probably and at the 
time I was looking for compute and storage all in one -and was advised not to 
basically.  So Hyperconvergence  is now a thing and want to try it out.  I was 
looking at the blog - up and running with ovirt 4.1, but it seems that is 
mostly for a lab setup.  I've got some real hardware I am going to try this out 
on as a PoC, 5 Dell R720s with 12 TB raw storage in each.  Just wondered if 
there was a guide to the hardware setup of my storage that I am missing?  
Should I configure the storage as hardware RAID before I start installing the 
Nodes?  I did setup glusterfs cluster 3 years ago as well but memory fails me 
on how I had the RAID controller setup if at all.

Thanks
___
Users mailing list -- users@ovirt.org  
To unsubscribe send an email to users-le...@ovirt.org 
 

 

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org

[ovirt-users] ovirt and gluster disk config

[william . dossett]

Hey,  been a few years since I looked at ovirt.   3 years probably and at the 
time I was looking for compute and storage all in one -and was advised not to 
basically.  So Hyperconvergence  is now a thing and want to try it out.  I was 
looking at the blog - up and running with ovirt 4.1, but it seems that is 
mostly for a lab setup.  I've got some real hardware I am going to try this out 
on as a PoC, 5 Dell R720s with 12 TB raw storage in each.  Just wondered if 
there was a guide to the hardware setup of my storage that I am missing?  
Should I configure the storage as hardware RAID before I start installing the 
Nodes?  I did setup glusterfs cluster 3 years ago as well but memory fails me 
on how I had the RAID controller setup if at all.

Thanks
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org