Re: [ovirt-users] [Gluster-users] GlusterFS performance with only one drive per host?
The bottleneck is definitely not the disk speed with glusterFS, no point of using SSD for bricks what so ever -v On Thu, Mar 22, 2018 at 6:01 AM, Sahina Bose wrote: > > > On Mon, Mar 19, 2018 at 5:57 PM, Jayme wrote: >> >> I'm spec'ing a new oVirt build using three Dell R720's w/ 256GB. I'm >> considering storage options. I don't have a requirement for high amounts of >> storage, I have a little over 1TB to store but want some overhead so I'm >> thinking 2TB of usable space would be sufficient. >> >> I've been doing some research on Micron 1100 2TB ssd's and they seem to >> offer a lot of value for the money. I'm considering using smaller cheaper >> SSDs for boot drives and using one 2TB micron SSD in each host for a >> glusterFS replica 3 setup (on the fence about using an arbiter, I like the >> extra redundancy replicate 3 will give me). >> >> My question is, would I see a performance hit using only one drive in each >> host with glusterFS or should I try to add more physical disks. Such as 6 >> 1TB drives instead of 3 2TB drives? > > > [Adding gluster-users for inputs here] > >> >> Also one other question. I've read that gluster can only be done in >> groups of three. Meaning you need 3, 6, or 9 hosts. Is this true? If I >> had an operational replicate 3 glusterFS setup and wanted to add more >> capacity I would have to add 3 more hosts, or is it possible for me to add a >> 4th host in to the mix for extra processing power down the road? > > > In oVirt, we support replica 3 or replica 3 with arbiter (where one of the 3 > bricks is a low storage arbiter brick). To expand storage, you would need to > add in multiples of 3 bricks. However if you only want to expand compute > capacity in your HC environment, you can add a 4th node. > >> >> Thanks! >> >> >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> > > > ___ > Gluster-users mailing list > gluster-us...@gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] [Gluster-users] GlusterFS performance with only one drive per host?
My take is that unless you have loads of data and are trying to optimize for cost/TB, HDDs are probably not the right choice. This is particularly true for random I/O workloads for which HDDs are really quite bad. I'd recommend a recent gluster release, and some tuning because the default settings are not optimized for performance. Some options to consider: client.event-threads server.event-threads cluster.choose-local performance.client-io-threads You can toggle the last two and see what works for you. You'd probably need to set event-threads to 4 or more. Ideally you'd tune some of the thread pools based on observed bottlenecks in collected stats. top (top -bHd 10 > top_threads.out.txt) is great for this. Using 6 small drives/bricks instead of 3 is also a good idea to reduce likelihood of rpc bottlenecks. There has been an effort to improve gluster performance over fast SSDs. Hence the recommendation to try with a recent release. You can also check in on some of the issues being worked on: https://github.com/gluster/glusterfs/issues/412 https://github.com/gluster/glusterfs/issues/410 -- Manoj On Sat, Mar 24, 2018 at 4:14 AM, Jayme wrote: > Do you feel that SSDs are worth the extra cost or am I better off using > regular HDDs? I'm looking for the best performance I can get with glusterFS > > On Fri, Mar 23, 2018 at 12:03 AM, Manoj Pillai wrote: > >> >> >> On Thu, Mar 22, 2018 at 3:31 PM, Sahina Bose wrote: >> >>> >>> >>> On Mon, Mar 19, 2018 at 5:57 PM, Jayme wrote: >>> I'm spec'ing a new oVirt build using three Dell R720's w/ 256GB. I'm considering storage options. I don't have a requirement for high amounts of storage, I have a little over 1TB to store but want some overhead so I'm thinking 2TB of usable space would be sufficient. I've been doing some research on Micron 1100 2TB ssd's and they seem to offer a lot of value for the money. I'm considering using smaller cheaper SSDs for boot drives and using one 2TB micron SSD in each host for a glusterFS replica 3 setup (on the fence about using an arbiter, I like the extra redundancy replicate 3 will give me). My question is, would I see a performance hit using only one drive in each host with glusterFS or should I try to add more physical disks. Such as 6 1TB drives instead of 3 2TB drives? >>> >> It is possible. With SSDs the rpc layer can become the bottleneck with >> some workloads, especially if there are not enough connections out to the >> server side. We had experimented with a multi-connection model for this >> reason: https://review.gluster.org/#/c/19133/. >> >> -- Manoj >> >>> >>> [Adding gluster-users for inputs here] >>> >>> Also one other question. I've read that gluster can only be done in groups of three. Meaning you need 3, 6, or 9 hosts. Is this true? If I had an operational replicate 3 glusterFS setup and wanted to add more capacity I would have to add 3 more hosts, or is it possible for me to add a 4th host in to the mix for extra processing power down the road? >>> >>> In oVirt, we support replica 3 or replica 3 with arbiter (where one of >>> the 3 bricks is a low storage arbiter brick). To expand storage, you would >>> need to add in multiples of 3 bricks. However if you only want to expand >>> compute capacity in your HC environment, you can add a 4th node. >>> >>> Thanks! ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users >>> >>> ___ >>> Gluster-users mailing list >>> gluster-us...@gluster.org >>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> >> > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] [Gluster-users] GlusterFS performance with only one drive per host?
On Thu, Mar 22, 2018 at 3:31 PM, Sahina Bose wrote: > > > On Mon, Mar 19, 2018 at 5:57 PM, Jayme wrote: > >> I'm spec'ing a new oVirt build using three Dell R720's w/ 256GB. I'm >> considering storage options. I don't have a requirement for high amounts >> of storage, I have a little over 1TB to store but want some overhead so I'm >> thinking 2TB of usable space would be sufficient. >> >> I've been doing some research on Micron 1100 2TB ssd's and they seem to >> offer a lot of value for the money. I'm considering using smaller cheaper >> SSDs for boot drives and using one 2TB micron SSD in each host for a >> glusterFS replica 3 setup (on the fence about using an arbiter, I like the >> extra redundancy replicate 3 will give me). >> >> My question is, would I see a performance hit using only one drive in >> each host with glusterFS or should I try to add more physical disks. Such >> as 6 1TB drives instead of 3 2TB drives? >> > It is possible. With SSDs the rpc layer can become the bottleneck with some workloads, especially if there are not enough connections out to the server side. We had experimented with a multi-connection model for this reason: https://review.gluster.org/#/c/19133/. -- Manoj > > [Adding gluster-users for inputs here] > > >> Also one other question. I've read that gluster can only be done in >> groups of three. Meaning you need 3, 6, or 9 hosts. Is this true? If I >> had an operational replicate 3 glusterFS setup and wanted to add more >> capacity I would have to add 3 more hosts, or is it possible for me to add >> a 4th host in to the mix for extra processing power down the road? >> > > In oVirt, we support replica 3 or replica 3 with arbiter (where one of the > 3 bricks is a low storage arbiter brick). To expand storage, you would need > to add in multiples of 3 bricks. However if you only want to expand compute > capacity in your HC environment, you can add a 4th node. > > >> Thanks! >> >> >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > > ___ > Gluster-users mailing list > gluster-us...@gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] [Gluster-users] GlusterFS performance with only one drive per host?
I would go with at least 4 HDDs per host in RAID 10. Then focus on network performance where bottleneck usualy is for gluster. On Sat, Mar 24, 2018, 00:44 Jayme wrote: > Do you feel that SSDs are worth the extra cost or am I better off using > regular HDDs? I'm looking for the best performance I can get with glusterFS > > On Fri, Mar 23, 2018 at 12:03 AM, Manoj Pillai wrote: > >> >> >> On Thu, Mar 22, 2018 at 3:31 PM, Sahina Bose wrote: >> >>> >>> >>> On Mon, Mar 19, 2018 at 5:57 PM, Jayme wrote: >>> I'm spec'ing a new oVirt build using three Dell R720's w/ 256GB. I'm considering storage options. I don't have a requirement for high amounts of storage, I have a little over 1TB to store but want some overhead so I'm thinking 2TB of usable space would be sufficient. I've been doing some research on Micron 1100 2TB ssd's and they seem to offer a lot of value for the money. I'm considering using smaller cheaper SSDs for boot drives and using one 2TB micron SSD in each host for a glusterFS replica 3 setup (on the fence about using an arbiter, I like the extra redundancy replicate 3 will give me). My question is, would I see a performance hit using only one drive in each host with glusterFS or should I try to add more physical disks. Such as 6 1TB drives instead of 3 2TB drives? >>> >> It is possible. With SSDs the rpc layer can become the bottleneck with >> some workloads, especially if there are not enough connections out to the >> server side. We had experimented with a multi-connection model for this >> reason: https://review.gluster.org/#/c/19133/. >> >> -- Manoj >> >>> >>> [Adding gluster-users for inputs here] >>> >>> Also one other question. I've read that gluster can only be done in groups of three. Meaning you need 3, 6, or 9 hosts. Is this true? If I had an operational replicate 3 glusterFS setup and wanted to add more capacity I would have to add 3 more hosts, or is it possible for me to add a 4th host in to the mix for extra processing power down the road? >>> >>> In oVirt, we support replica 3 or replica 3 with arbiter (where one of >>> the 3 bricks is a low storage arbiter brick). To expand storage, you would >>> need to add in multiples of 3 bricks. However if you only want to expand >>> compute capacity in your HC environment, you can add a 4th node. >>> >>> Thanks! ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users >>> >>> ___ >>> Gluster-users mailing list >>> gluster-us...@gluster.org >>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> >> > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] [Gluster-users] GlusterFS performance with only one drive per host?
Do you feel that SSDs are worth the extra cost or am I better off using regular HDDs? I'm looking for the best performance I can get with glusterFS On Fri, Mar 23, 2018 at 12:03 AM, Manoj Pillai wrote: > > > On Thu, Mar 22, 2018 at 3:31 PM, Sahina Bose wrote: > >> >> >> On Mon, Mar 19, 2018 at 5:57 PM, Jayme wrote: >> >>> I'm spec'ing a new oVirt build using three Dell R720's w/ 256GB. I'm >>> considering storage options. I don't have a requirement for high amounts >>> of storage, I have a little over 1TB to store but want some overhead so I'm >>> thinking 2TB of usable space would be sufficient. >>> >>> I've been doing some research on Micron 1100 2TB ssd's and they seem to >>> offer a lot of value for the money. I'm considering using smaller cheaper >>> SSDs for boot drives and using one 2TB micron SSD in each host for a >>> glusterFS replica 3 setup (on the fence about using an arbiter, I like the >>> extra redundancy replicate 3 will give me). >>> >>> My question is, would I see a performance hit using only one drive in >>> each host with glusterFS or should I try to add more physical disks. Such >>> as 6 1TB drives instead of 3 2TB drives? >>> >> > It is possible. With SSDs the rpc layer can become the bottleneck with > some workloads, especially if there are not enough connections out to the > server side. We had experimented with a multi-connection model for this > reason: https://review.gluster.org/#/c/19133/. > > -- Manoj > >> >> [Adding gluster-users for inputs here] >> >> >>> Also one other question. I've read that gluster can only be done in >>> groups of three. Meaning you need 3, 6, or 9 hosts. Is this true? If I >>> had an operational replicate 3 glusterFS setup and wanted to add more >>> capacity I would have to add 3 more hosts, or is it possible for me to add >>> a 4th host in to the mix for extra processing power down the road? >>> >> >> In oVirt, we support replica 3 or replica 3 with arbiter (where one of >> the 3 bricks is a low storage arbiter brick). To expand storage, you would >> need to add in multiples of 3 bricks. However if you only want to expand >> compute capacity in your HC environment, you can add a 4th node. >> >> >>> Thanks! >>> >>> >>> ___ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> >> ___ >> Gluster-users mailing list >> gluster-us...@gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users