Re: [Gluster-users] Production cluster planning

On 10/27/2016 12:06 AM, Gandalf Corvotempesta wrote: 2016-10-26 23:38 GMT+02:00 Joe Julian : Just do the reliability calculations and engineer a storage system to meet (exceed) your obligations within the available budget.

Re: [Gluster-users] Production cluster planning

2016-10-26 23:38 GMT+02:00 Joe Julian : > Just do the reliability calculations and engineer a storage system to meet > (exceed) your obligations within the available budget. > http://www.eventhelix.com/realtimemantra/faulthandling/system_reliability_availability.htm > This

Re: [Gluster-users] Production cluster planning

2016-10-26 23:38 GMT+02:00 Joe Julian : > Quickly = MTTR is within tolerances to continue to meet SLA. It's just math. Obviously yes. But in the real world, you can have the best SLAs in the world, but if you loose data, you loose customers. > As for a dedicated heal

Re: [Gluster-users] Production cluster planning

2016-10-27 0:14 GMT+02:00 Joe Julian : > For now I can say that gluster performs better and has a much better > worst-case resolution. If everything else goes to hell, I have disks with > files on them that I can recover on a laptop if I have to. Totally agree. > Of

Re: [Gluster-users] Production cluster planning

On 10/26/2016 03:42 PM, Lindsay Mathieson wrote: On 27/10/2016 8:14 AM, Joe Julian wrote: To be fair, though, I can't blame ceph. We had a cascading hardware failure with those storage trays. Even still, if it had been gluster - I would have had files on disks. Ouch :( In that regard how

Re: [Gluster-users] Production cluster planning

On 27/10/2016 8:14 AM, Joe Julian wrote: To be fair, though, I can't blame ceph. We had a cascading hardware failure with those storage trays. Even still, if it had been gluster - I would have had files on disks. Ouch :( In that regard how do you view sharding? why not as simple as pulling

Re: [Gluster-users] Production cluster planning

On 10/26/2016 02:54 PM, Lindsay Mathieson wrote: Maybe a controversial question (and hopefully not trolling), but any particularly reason you choose gluster over ceph for these larger setups Joe? For myself, gluster is much easier to manage and provides better performance on my small

Re: [Gluster-users] Production cluster planning

Maybe a controversial question (and hopefully not trolling), but any particularly reason you choose gluster over ceph for these larger setups Joe? For myself, gluster is much easier to manage and provides better performance on my small non-enterprise setup, plus it plays nice with zfs. But

Re: [Gluster-users] Production cluster planning

On 10/26/2016 02:12 PM, Gandalf Corvotempesta wrote: 2016-10-26 23:07 GMT+02:00 Joe Julian : And yes, they can fail, but 20TB is small enough to heal pretty quickly. 20TB small enough to build quickly? On which network? Gluster doesn't have a dedicated cluster network,

Re: [Gluster-users] Production cluster planning

2016-10-26 23:09 GMT+02:00 Joe Julian : > Open Compute WiWynn Knox trays. I don't recommend them but they are pretty. > https://goo.gl/photos/tmkRE58xKKaWKdL96 What are you hosting on that huge cluster? 10GB network I suppose. ___

Re: [Gluster-users] Production cluster planning

2016-10-26 23:07 GMT+02:00 Joe Julian : > And yes, they can fail, but 20TB is small enough to heal pretty quickly. 20TB small enough to build quickly? On which network? Gluster doesn't have a dedicated cluster network, if the cluster is being hevily accessed, the healing

Re: [Gluster-users] Production cluster planning

On 10/26/2016 02:06 PM, Gandalf Corvotempesta wrote: 2016-10-26 23:04 GMT+02:00 Joe Julian : I just add enough disks to saturate (and I don't like zfs, personally) per-brick. So with 30 disks on a server, I typically do 5-disk raid-0 and create 6 bricks per server. 30

Re: [Gluster-users] Production cluster planning

On 10/26/2016 02:06 PM, Gandalf Corvotempesta wrote: 2016-10-26 23:04 GMT+02:00 Joe Julian : I just add enough disks to saturate (and I don't like zfs, personally) per-brick. So with 30 disks on a server, I typically do 5-disk raid-0 and create 6 bricks per server. 30

Re: [Gluster-users] Production cluster planning

On 10/26/2016 02:04 PM, Joe Julian wrote: On 10/26/2016 02:02 PM, Gandalf Corvotempesta wrote: 2016-10-26 22:59 GMT+02:00 Joe Julian : Personally, I prefer raid0 bricks just to get the throughput to saturate my network, then I use replicate to meet my availability

Re: [Gluster-users] Production cluster planning

2016-10-26 23:04 GMT+02:00 Joe Julian : > I just add enough disks to saturate (and I don't like zfs, personally) > per-brick. So with 30 disks on a server, I typically do 5-disk raid-0 and > create 6 bricks per server. 30 disks per server? which chassis are you using? Why

Re: [Gluster-users] Production cluster planning

On 27/10/2016 6:59 AM, Joe Julian wrote: Personally, I prefer raid0 bricks just to get the throughput to saturate my network, then I use replicate to meet my availability requirements (typically replica 3). My network is the limiting factor already :( Only 1G * 3 Bond. Cheap and nasty D-Link

Re: [Gluster-users] Production cluster planning

On 10/26/2016 02:02 PM, Gandalf Corvotempesta wrote: 2016-10-26 22:59 GMT+02:00 Joe Julian : Personally, I prefer raid0 bricks just to get the throughput to saturate my network, then I use replicate to meet my availability requirements (typically replica 3). Isn't the

Re: [Gluster-users] Production cluster planning

On 27/10/2016 6:58 AM, mabi wrote: Sorry yes I meant vmstat, I was doing too much ionice/iostat today ;) right now its averaging at 45000. Low load though. -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org

Re: [Gluster-users] Production cluster planning

2016-10-26 22:59 GMT+02:00 Joe Julian : > Personally, I prefer raid0 bricks just to get the throughput to saturate my > network, then I use replicate to meet my availability requirements > (typically replica 3). Isn't the ZFS cache on SSD enough to saturate the network?

Re: [Gluster-users] Production cluster planning

On 27/10/2016 6:56 AM, Gandalf Corvotempesta wrote: Velociraptors: are still around ? I heard that were EOL a couple of years ago. Legacy hardware :) I must admit they last really well. -- Lindsay Mathieson ___ Gluster-users mailing list

Re: [Gluster-users] Production cluster planning

On 10/26/2016 01:56 PM, Gandalf Corvotempesta wrote: 2016-10-26 22:31 GMT+02:00 Lindsay Mathieson : Yah, RAID10. - Two nodes with 4 WD 3TB RED I really hate RAID10. Personally, I prefer raid0 bricks just to get the throughput to saturate my network, then I use

Re: [Gluster-users] Production cluster planning

Sorry yes I meant vmstat, I was doing too much ionice/iostat today ;) Original Message Subject: Re: [Gluster-users] Production cluster planning Local Time: October 26, 2016 10:56 PM UTC Time: October 26, 2016 8:56 PM From: lindsay.mathie...@gmail.com To: gluster-users

Re: [Gluster-users] Production cluster planning

2016-10-26 22:31 GMT+02:00 Lindsay Mathieson : > Yah, RAID10. > > - Two nodes with 4 WD 3TB RED I really hate RAID10. I'm evaluating 2 RAIZ2 on each gluster node (12 disks: 6+6 on each RAIDZ2) or one huge RAIDZ3 with 12 disks. The biggest drawback with RAIDZ is that

Re: [Gluster-users] Production cluster planning

On 27/10/2016 6:35 AM, mabi wrote: I was wondering with your setup you mention, how high are your context switches? I mean what is your typical average context switch and what are your highest context switch peeks (as seen in iostat). Wouldn't that be vmstat? -- Lindsay Mathieson

Re: [Gluster-users] Production cluster planning

I was wondering with your setup you mention, how high are your context switches? I mean what is your typical average context switch and what are your highest context switch peeks (as seen in iostat). Best, M. Original Message Subject: Re: [Gluster-users] Production

Re: [Gluster-users] Production cluster planning

2016-10-05 23:48 GMT+02:00 Lindsay Mathieson : > Its enough? I also run 10 windows VM's per node. > > > My servers typically run at 4-6% max ioload. They idle under 1% Are you using any ZFS RAID on your servers? ___

Re: [Gluster-users] Production cluster planning

On 6/10/2016 6:37 AM, Gandalf Corvotempesta wrote: Only 8GB ? Why ? Its enough? I also run 10 windows VM's per node. My servers typically run at 4-6% max ioload. They idle under 1% -- Lindsay Mathieson ___ Gluster-users mailing list

Re: [Gluster-users] Production cluster planning

2016-10-05 22:35 GMT+02:00 Lindsay Mathieson : > 64Gb RAM in each server, 8GB reversed for ZFS. Only 8GB ? Why ? ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Production cluster planning

On 6/10/2016 6:20 AM, Gandalf Corvotempesta wrote: > L2ARC depends on your workload. For me its not useful - VM hosting on a sharded volume, never got better than 6% cache hits. The vast majority of hits were via ARC (memory). ZFS seems to be really good at that :) > How many ram do you

Re: [Gluster-users] Production cluster planning

Il 05 ott 2016 10:12 PM, "Lindsay Mathieson" ha scritto: > > L2ARC depends on your workload. For me its not useful - VM hosting on a sharded volume, never got better than 6% cache hits. The vast majority of hits were via ARC (memory). ZFS seems to be really good at

Re: [Gluster-users] Production cluster planning

On 6/10/2016 4:29 AM, Gandalf Corvotempesta wrote: > I was thinking about creating one or more raidz2 to use as bricks, with 2 ssd. One small partition on these ssd would be used as a mirrored SLOG and the other 2 would be used as standalone arc cache. will this worth the use of SSD or would

Re: [Gluster-users] Production cluster planning

On September 30, 2016 1:46:31 PM GMT+02:00, Gandalf Corvotempesta wrote: > >As suggestion for gluster developers: if ZFS is considered stable it >could >be used as default (replacing xfs) and many features that zfs already >has >could be removed from gluster

Re: [Gluster-users] Production cluster planning

On Wed, Oct 5, 2016 at 2:14 PM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > 2016-10-05 20:50 GMT+02:00 David Gossage : > > The mirrored slog will be useful. Depending on what you put on the pool > > l2arc may not get used much. I removed mine

Re: [Gluster-users] Production cluster planning

2016-10-05 20:50 GMT+02:00 David Gossage : > The mirrored slog will be useful. Depending on what you put on the pool > l2arc may not get used much. I removed mine as it got such a low hit rate > serving VM's. I'll use shards. The most accessed shard isn't cached in

Re: [Gluster-users] Production cluster planning

On Wed, Oct 5, 2016 at 1:29 PM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > Il 30 set 2016 1:46 PM, "Gandalf Corvotempesta" < > gandalf.corvotempe...@gmail.com> ha scritto: > > > I was thinking about creating one or more raidz2 to use as bricks, with > 2 ssd. One small

Re: [Gluster-users] Production cluster planning

Il 30 set 2016 1:46 PM, "Gandalf Corvotempesta" < gandalf.corvotempe...@gmail.com> ha scritto: > I was thinking about creating one or more raidz2 to use as bricks, with 2 ssd. One small partition on these ssd would be used as a mirrored SLOG and the other 2 would be used as standalone arc cache.

Re: [Gluster-users] Production cluster planning

On 1/10/2016 4:15 AM, mabi wrote: The data will not be in "any" state as you mention or please define what you mean by "any". In the worst case you will just loose 5 seconds of data that's all as far as I understand. By "Any" state I mean *Any*, you have no way of predicting how much data

Re: [Gluster-users] Production cluster planning

Sorry the link is missing in my previous post: https://groups.google.com/a/zfsonlinux.org/d/msg/zfs-discuss/OI5dchl7d_8/vLRMZgJGYUoJ Original Message Subject: Re: [Gluster-users] Production cluster planning Local Time: September 30, 2016 8:15 PM UTC Time: September 30

Re: [Gluster-users] Production cluster planning

y it all boils down to this specific Original Message ---- Subject: Re: [Gluster-users] Production cluster planning Local Time: September 30, 2016 12:41 PM UTC Time: September 30, 2016 10:41 AM From: lindsay.mathie...@gmail.com To: mabi <m...@protonmail.ch>, Gluster Users <gluster

Re: [Gluster-users] Production cluster planning

2016-09-30 12:41 GMT+02:00 Lindsay Mathieson : > Your missing what he said - *ZFS* will not be corrupted but the data written > could be in any state, in this case the gluster filesystem data and meta > data. To have one ndoe in a cluster out of sync with out the

Re: [Gluster-users] Production cluster planning

On 29/09/2016 4:32 AM, mabi wrote: hat's not correct. There is no risk of corruption using "sync=disabled". In the worst case you just end up with old data but no corruption. See the following comment from a master of ZFS (Aaron Toponce):

Re: [Gluster-users] Production cluster planning

Il 30 set 2016 11:35, "mabi" ha scritto: > > That's not correct. There is no risk of corruption using "sync=disabled". In the worst case you just end up with old data but no corruption. See the following comment from a master of ZFS (Aaron Toponce): > >

Re: [Gluster-users] Production cluster planning

ubject: Re: [Gluster-users] Production cluster planning Local Time: September 26, 2016 11:08 PM UTC Time: September 26, 2016 9:08 PM From: lindsay.mathie...@gmail.com To: gluster-users@gluster.org On 27/09/2016 4:13 AM, mabi wrote: > I would also say do not forget to set "sync=disabled"

Re: [Gluster-users] Production cluster planning

2016-09-26 23:08 GMT+02:00 Lindsay Mathieson : > I wouldn't be doing that - very high risk of gluster corruption in the event > of power loss or server crash. Up to 5 seconds of writes could be lost that > way. > > If writes aren't fast enough I'd add a SSD partition

Re: [Gluster-users] Production cluster planning

On 27/09/2016 4:13 AM, mabi wrote: I would also say do not forget to set "sync=disabled". I wouldn't be doing that - very high risk of gluster corruption in the event of power loss or server crash. Up to 5 seconds of writes could be lost that way. If writes aren't fast enough I'd add a

Re: [Gluster-users] Production cluster planning

I would also say do not forget to set "sync=disabled". Original Message Subject: Re: [Gluster-users] Production cluster planning Local Time: September 26, 2016 7:59 PM UTC Time: September 26, 2016 5:59 PM From: jlawre...@squaretrade.com To: Lindsay Mathieson <l

Re: [Gluster-users] Production cluster planning

> On Sep 26, 2016, at 4:26 AM, Lindsay Mathieson > wrote: > > On 26/09/2016 8:18 PM, Gandalf Corvotempesta wrote: >> No one ? >> And what about gluster on ZFS? Is that fully supported ? > > I certainly hope so because I'm running a Replica 3 production cluster on

Re: [Gluster-users] Production cluster planning

Hi, Red Hat only supports XFS for some reason: https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Installation_Guide/sect-Prerequisites1.html -- Dmitry Glushenok Jet Infosystems > 26 сент. 2016 г., в 14:26, Lindsay Mathieson > написал(а): > >

Re: [Gluster-users] Production cluster planning

On 26/09/2016 8:18 PM, Gandalf Corvotempesta wrote: No one ? And what about gluster on ZFS? Is that fully supported ? I certainly hope so because I'm running a Replica 3 production cluster on ZFS :) -- Lindsay Mathieson ___ Gluster-users mailing

Re: [Gluster-users] Production cluster planning

2016-09-26 12:22 GMT+02:00 David Gossage : > I don't know that they test on zfs, but when I had zfs related issues with > some changes they were very accommodating in assisting the testing and bug > fix when I opened a bugzilla report. I'm asking this because I would

Re: [Gluster-users] Production cluster planning

On Mon, Sep 26, 2016 at 5:18 AM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > 2016-09-21 1:04 GMT+02:00 Pranith Kumar Karampuri : > > I am not sure about this one, I was waiting for someone else to respond > on > > this point. > > No one ? > And what

Re: [Gluster-users] Production cluster planning

2016-09-21 1:04 GMT+02:00 Pranith Kumar Karampuri : > I am not sure about this one, I was waiting for someone else to respond on > this point. No one ? And what about gluster on ZFS? Is that fully supported ? ___ Gluster-users

Re: [Gluster-users] Production cluster planning

On Tue, Sep 20, 2016 at 5:44 PM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > 2016-09-20 12:06 GMT+02:00 Pranith Kumar Karampuri : > > Replica 3 is more costly compared to Arbiter. If you are comfortable with > > replica-3 that is the best option. > > And

Re: [Gluster-users] Production cluster planning

2016-09-20 12:06 GMT+02:00 Pranith Kumar Karampuri : > Replica 3 is more costly compared to Arbiter. If you are comfortable with > replica-3 that is the best option. And what about raid and ssd tiering? Which is the most common and stable configuration available?

Re: [Gluster-users] Production cluster planning

On Tue, Sep 20, 2016 at 3:31 PM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > 2016-09-20 11:57 GMT+02:00 Pranith Kumar Karampuri : > > I would suggest to use separate volumes for this, because the > optimizations > > for VM image store on the volume would

Re: [Gluster-users] Production cluster planning

2016-09-20 11:57 GMT+02:00 Pranith Kumar Karampuri : > I would suggest to use separate volumes for this, because the optimizations > for VM image store on the volume would slow things down for webservers kind > of workload IMHO. Most of the perf xlators should be disabled in

Re: [Gluster-users] Production cluster planning

On Tue, Sep 20, 2016 at 3:24 PM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > 2016-09-20 10:28 GMT+02:00 Pranith Kumar Karampuri : > > Are you willing to take VM snapshots at the right times? > > Yes, sure. > > In addition, the same cluster would also be

Re: [Gluster-users] Production cluster planning

2016-09-20 10:28 GMT+02:00 Pranith Kumar Karampuri : > Are you willing to take VM snapshots at the right times? Yes, sure. In addition, the same cluster would also be used to store files shared between webservers, like the uploaded images and so on. > Please don't use EC

Re: [Gluster-users] Production cluster planning

On Fri, Sep 16, 2016 at 10:23 PM, Gandalf Corvotempesta < gandalf.corvotempe...@gmail.com> wrote: > Next year i'll start with our first production cluster. > I'll put on that many VMs images (XenServer, ProxMox, ...). > > Currently I have 3 SuperMicro 6028R-E1CR12T to be used as storage nodes. >

Re: [Gluster-users] Production cluster planning

No one? Il 16 set 2016 18:53, "Gandalf Corvotempesta" < gandalf.corvotempe...@gmail.com> ha scritto: > Next year i'll start with our first production cluster. > I'll put on that many VMs images (XenServer, ProxMox, ...). > > Currently I have 3 SuperMicro 6028R-E1CR12T to be used as storage

[Gluster-users] Production cluster planning

Next year i'll start with our first production cluster. I'll put on that many VMs images (XenServer, ProxMox, ...). Currently I have 3 SuperMicro 6028R-E1CR12T to be used as storage nodes. I'll put 2 more 10GbT cards on each. Primary goal is to have MAXIMUM data redundancy and protection. we