Re: [Gluster-users] multi petabyte gluster dispersed for archival?

[Douglas Duckworth]

Replication would be better yes but HA isn't a hard requirement whereas the 
most likely loss of a brick would be power.  In that case we could stop the 
entire file system then bring the brick back up should users complain about 
poor I/O performance.

Could you share more about your configuration at that time?  What CPUs were you 
running on bricks, number of spindles per brick, etc?


--
Thanks,

Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit<https://scu.med.cornell.edu/>
Weill Cornell Medicine
E: d...@med.cornell.edu
O: 212-746-6305
F: 212-746-8690


From: Serkan Çoban 
Sent: Thursday, February 13, 2020 12:38 PM
To: Douglas Duckworth 
Cc: gluster-users@gluster.org 
Subject: [EXTERNAL] Re: [Gluster-users] multi petabyte gluster dispersed for 
archival?

Do not use EC with small files. You cannot tolerate losing a 300TB
brick, reconstruction will take ages. When I was using glusterfs
reconstruction speed of ec was 10-15MB/sec. If you do not loose bricks
you will be ok.

On Thu, Feb 13, 2020 at 7:38 PM Douglas Duckworth
 wrote:
>
> Hello
>
> I am thinking of building a Gluster file system for archival data.  Initially 
> it will start as 6 brick dispersed volume then expand to distributed 
> dispersed as we increase capacity.
>
> Since metadata in Gluster isn't centralized it will eventually not perform 
> well at scale.  So I am wondering if anyone can help identify that point?  
> Ceph can scale to extremely high levels though the complexity required for 
> management seems much greater than Gluster.
>
> The first six bricks would be a little over 2PB of raw space.  Each server 
> will have 24 7200 RPM NL-SAS drives sans RAID.  I estimate we would max out 
> at about 100 million files within these first six servers, though that can be 
> reduced by having users tar their small files before writing to Gluster.   
> I/O patterns would be sequential upon initial copy with very infrequent reads 
> thereafter.  Given the demands of erasure coding, especially if we lose a 
> brick, the CPUs will be high thread count AMD Rome.  The back-end network 
> would be EDR Infiniband, so I will mount via RDMA, while all bricks will be 
> leaf local.
>
> Given these variables can anyone say whether Gluster would be able to operate 
> at this level of metadata and continue to scale?  If so where could it break, 
> 4PB, 12PB, with that being defined as I/O, with all bricks still online, 
> breaking down dramatically?
>
> Thank you!
> Doug
>
> --
> Thanks,
>
> Douglas Duckworth, MSc, LFCS
> HPC System Administrator
> Scientific Computing Unit
> Weill Cornell Medicine
> E: d...@med.cornell.edu
> O: 212-746-6305
> F: 212-746-8690
>
> 
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bluejeans.com_441850968=DwIFaQ=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw=SsvW0KsQAhI5SQf6z4WQde56D5y5zBm3wJkCyyiVj6E=-tl_YiEBCYUEm7rzTkbvmTck0LsAurEd9DJaq8v5-fc=
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__bluejeans.com_441850968=DwIFaQ=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw=SsvW0KsQAhI5SQf6z4WQde56D5y5zBm3wJkCyyiVj6E=-tl_YiEBCYUEm7rzTkbvmTck0LsAurEd9DJaq8v5-fc=
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gluster.org_mailman_listinfo_gluster-2Dusers=DwIFaQ=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw=SsvW0KsQAhI5SQf6z4WQde56D5y5zBm3wJkCyyiVj6E=i7jvEHb-wZksUurCWq828kigRsSxfrAiNWxT7ORcgFs=


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] multi petabyte gluster dispersed for archival?

[Serkan Çoban]

I was using EC configuration 16+4 with 40 servers, each server has
68x10TB JBOD disks.
Glusterfs is mounted to 1000 hadoop datanodes, we were using glusterfs
as hadoop archive.
When a disk fails we did not loose write speed but read speed slows
down too much. I used glusterfs at the edge and it served its purpose.
Beware that metadata operations will be much slower as ec size increases.

Now hdfs also has EC so we replaced it with hdfs.
Our servers had 2x12 core cpus with 256GB RAM each with 2x10G bonded
interfaces. CPU is used heavily during reconstruction.

On Thu, Feb 13, 2020 at 8:44 PM Douglas Duckworth
 wrote:
>
> Replication would be better yes but HA isn't a hard requirement whereas the 
> most likely loss of a brick would be power.  In that case we could stop the 
> entire file system then bring the brick back up should users complain about 
> poor I/O performance.
>
> Could you share more about your configuration at that time?  What CPUs were 
> you running on bricks, number of spindles per brick, etc?
>
> --
> Thanks,
>
> Douglas Duckworth, MSc, LFCS
> HPC System Administrator
> Scientific Computing Unit
> Weill Cornell Medicine
> E: d...@med.cornell.edu
> O: 212-746-6305
> F: 212-746-8690
>
> 
> From: Serkan Çoban 
> Sent: Thursday, February 13, 2020 12:38 PM
> To: Douglas Duckworth 
> Cc: gluster-users@gluster.org 
> Subject: [EXTERNAL] Re: [Gluster-users] multi petabyte gluster dispersed for 
> archival?
>
> Do not use EC with small files. You cannot tolerate losing a 300TB
> brick, reconstruction will take ages. When I was using glusterfs
> reconstruction speed of ec was 10-15MB/sec. If you do not loose bricks
> you will be ok.
>
> On Thu, Feb 13, 2020 at 7:38 PM Douglas Duckworth
>  wrote:
> >
> > Hello
> >
> > I am thinking of building a Gluster file system for archival data.  
> > Initially it will start as 6 brick dispersed volume then expand to 
> > distributed dispersed as we increase capacity.
> >
> > Since metadata in Gluster isn't centralized it will eventually not perform 
> > well at scale.  So I am wondering if anyone can help identify that point?  
> > Ceph can scale to extremely high levels though the complexity required for 
> > management seems much greater than Gluster.
> >
> > The first six bricks would be a little over 2PB of raw space.  Each server 
> > will have 24 7200 RPM NL-SAS drives sans RAID.  I estimate we would max out 
> > at about 100 million files within these first six servers, though that can 
> > be reduced by having users tar their small files before writing to Gluster. 
> >   I/O patterns would be sequential upon initial copy with very infrequent 
> > reads thereafter.  Given the demands of erasure coding, especially if we 
> > lose a brick, the CPUs will be high thread count AMD Rome.  The back-end 
> > network would be EDR Infiniband, so I will mount via RDMA, while all bricks 
> > will be leaf local.
> >
> > Given these variables can anyone say whether Gluster would be able to 
> > operate at this level of metadata and continue to scale?  If so where could 
> > it break, 4PB, 12PB, with that being defined as I/O, with all bricks still 
> > online, breaking down dramatically?
> >
> > Thank you!
> > Doug
> >
> > --
> > Thanks,
> >
> > Douglas Duckworth, MSc, LFCS
> > HPC System Administrator
> > Scientific Computing Unit
> > Weill Cornell Medicine
> > E: d...@med.cornell.edu
> > O: 212-746-6305
> > F: 212-746-8690
> >
> > 
> >
> > Community Meeting Calendar:
> >
> > APAC Schedule -
> > Every 2nd and 4th Tuesday at 11:30 AM IST
> > Bridge: 
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__bluejeans.com_441850968=DwIFaQ=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw=SsvW0KsQAhI5SQf6z4WQde56D5y5zBm3wJkCyyiVj6E=-tl_YiEBCYUEm7rzTkbvmTck0LsAurEd9DJaq8v5-fc=
> >
> > NA/EMEA Schedule -
> > Every 1st and 3rd Tuesday at 01:00 PM EDT
> > Bridge: 
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__bluejeans.com_441850968=DwIFaQ=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw=SsvW0KsQAhI5SQf6z4WQde56D5y5zBm3wJkCyyiVj6E=-tl_YiEBCYUEm7rzTkbvmTck0LsAurEd9DJaq8v5-fc=
> >
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gluster.org_mailman_listinfo_gluster-2Dusers=DwIFaQ=lb62iw4YL4RFalcE2hQUQealT9-RXrryqt9KZX2qu2s=2Fzhh_78OGspKQpl_e-CbhH6xUjnRkaqPFUS2wTJ2cw=SsvW0KsQAhI5SQf6z4WQde56D5y5zBm3wJkCyyiVj6E=i7jvEHb-wZksUurCWq828kigRsSxfrAiNWxT7ORcgFs=


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] multi petabyte gluster dispersed for archival?

[Serkan Çoban]

Do not use EC with small files. You cannot tolerate losing a 300TB
brick, reconstruction will take ages. When I was using glusterfs
reconstruction speed of ec was 10-15MB/sec. If you do not loose bricks
you will be ok.

On Thu, Feb 13, 2020 at 7:38 PM Douglas Duckworth
 wrote:
>
> Hello
>
> I am thinking of building a Gluster file system for archival data.  Initially 
> it will start as 6 brick dispersed volume then expand to distributed 
> dispersed as we increase capacity.
>
> Since metadata in Gluster isn't centralized it will eventually not perform 
> well at scale.  So I am wondering if anyone can help identify that point?  
> Ceph can scale to extremely high levels though the complexity required for 
> management seems much greater than Gluster.
>
> The first six bricks would be a little over 2PB of raw space.  Each server 
> will have 24 7200 RPM NL-SAS drives sans RAID.  I estimate we would max out 
> at about 100 million files within these first six servers, though that can be 
> reduced by having users tar their small files before writing to Gluster.   
> I/O patterns would be sequential upon initial copy with very infrequent reads 
> thereafter.  Given the demands of erasure coding, especially if we lose a 
> brick, the CPUs will be high thread count AMD Rome.  The back-end network 
> would be EDR Infiniband, so I will mount via RDMA, while all bricks will be 
> leaf local.
>
> Given these variables can anyone say whether Gluster would be able to operate 
> at this level of metadata and continue to scale?  If so where could it break, 
> 4PB, 12PB, with that being defined as I/O, with all bricks still online, 
> breaking down dramatically?
>
> Thank you!
> Doug
>
> --
> Thanks,
>
> Douglas Duckworth, MSc, LFCS
> HPC System Administrator
> Scientific Computing Unit
> Weill Cornell Medicine
> E: d...@med.cornell.edu
> O: 212-746-6305
> F: 212-746-8690
>
> 
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] multi petabyte gluster dispersed for archival?

[Douglas Duckworth]

Hello

I am thinking of building a Gluster file system for archival data.  Initially 
it will start as 6 brick dispersed volume then expand to distributed dispersed 
as we increase capacity.

Since metadata in Gluster isn't centralized it will eventually not perform well 
at scale.  So I am wondering if anyone can help identify that point?  Ceph can 
scale to extremely high levels though the complexity required for management 
seems much greater than Gluster.

The first six bricks would be a little over 2PB of raw space.  Each server will 
have 24 7200 RPM NL-SAS drives sans RAID.  I estimate we would max out at about 
100 million files within these first six servers, though that can be reduced by 
having users tar their small files before writing to Gluster.   I/O patterns 
would be sequential upon initial copy with very infrequent reads thereafter.  
Given the demands of erasure coding, especially if we lose a brick, the CPUs 
will be high thread count AMD Rome.  The back-end network would be EDR 
Infiniband, so I will mount via RDMA, while all bricks will be leaf local.

Given these variables can anyone say whether Gluster would be able to operate 
at this level of metadata and continue to scale?  If so where could it break, 
4PB, 12PB, with that being defined as I/O, with all bricks still online, 
breaking down dramatically?

Thank you!
Doug


--
Thanks,

Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit
Weill Cornell Medicine
E: d...@med.cornell.edu
O: 212-746-6305
F: 212-746-8690


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Multi petabyte gluster

[Alastair Neil]

I can ask our other engineer but I don't have those figues.

-Alastair


On 30 June 2017 at 13:52, Serkan Çoban <cobanser...@gmail.com> wrote:

> Did you test healing by increasing disperse.shd-max-threads?
> What is your heal times per brick now?
>
> On Fri, Jun 30, 2017 at 8:01 PM, Alastair Neil <ajneil.t...@gmail.com>
> wrote:
> > We are using 3.10 and have a 7 PB cluster.  We decided against 16+3 as
> the
> > rebuild time are bottlenecked by matrix operations which scale as the
> square
> > of the number of data stripes.  There are some savings because of larger
> > data chunks but we ended up using 8+3 and heal times are about half
> compared
> > to 16+3.
> >
> > -Alastair
> >
> > On 30 June 2017 at 02:22, Serkan Çoban <cobanser...@gmail.com> wrote:
> >>
> >> >Thanks for the reply. We will mainly use this for archival - near-cold
> >> > storage.
> >> Archival usage is good for EC
> >>
> >> >Anything, from your experience, to keep in mind while planning large
> >> > installations?
> >> I am using 3.7.11 and only problem is slow rebuild time when a disk
> >> fails. It takes 8 days to heal a 8TB disk.(This might be related with
> >> my EC configuration 16+4)
> >> 3.9+ versions has some improvements about this but I cannot test them
> >> yet...
> >>
> >> On Thu, Jun 29, 2017 at 2:49 PM, jkiebzak <jkieb...@gmail.com> wrote:
> >> > Thanks for the reply. We will mainly use this for archival - near-cold
> >> > storage.
> >> >
> >> >
> >> > Anything, from your experience, to keep in mind while planning large
> >> > installations?
> >> >
> >> >
> >> > Sent from my Verizon, Samsung Galaxy smartphone
> >> >
> >> >  Original message 
> >> > From: Serkan Çoban <cobanser...@gmail.com>
> >> > Date: 6/29/17 4:39 AM (GMT-05:00)
> >> > To: Jason Kiebzak <jkieb...@gmail.com>
> >> > Cc: Gluster Users <gluster-users@gluster.org>
> >> > Subject: Re: [Gluster-users] Multi petabyte gluster
> >> >
> >> > I am currently using 10PB single volume without problems. 40PB is on
> >> > the way. EC is working fine.
> >> > You need to plan ahead with large installations like this. Do complete
> >> > workload tests and make sure your use case is suitable for EC.
> >> >
> >> >
> >> > On Wed, Jun 28, 2017 at 11:18 PM, Jason Kiebzak <jkieb...@gmail.com>
> >> > wrote:
> >> >> Has anyone scaled to a multi petabyte gluster setup? How well does
> >> >> erasure
> >> >> code do with such a large setup?
> >> >>
> >> >> Thanks
> >> >>
> >> >> ___
> >> >> Gluster-users mailing list
> >> >> Gluster-users@gluster.org
> >> >> http://lists.gluster.org/mailman/listinfo/gluster-users
> >> ___
> >> Gluster-users mailing list
> >> Gluster-users@gluster.org
> >> http://lists.gluster.org/mailman/listinfo/gluster-users
> >
> >
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Multi petabyte gluster

[Serkan Çoban]

Did you test healing by increasing disperse.shd-max-threads?
What is your heal times per brick now?

On Fri, Jun 30, 2017 at 8:01 PM, Alastair Neil <ajneil.t...@gmail.com> wrote:
> We are using 3.10 and have a 7 PB cluster.  We decided against 16+3 as the
> rebuild time are bottlenecked by matrix operations which scale as the square
> of the number of data stripes.  There are some savings because of larger
> data chunks but we ended up using 8+3 and heal times are about half compared
> to 16+3.
>
> -Alastair
>
> On 30 June 2017 at 02:22, Serkan Çoban <cobanser...@gmail.com> wrote:
>>
>> >Thanks for the reply. We will mainly use this for archival - near-cold
>> > storage.
>> Archival usage is good for EC
>>
>> >Anything, from your experience, to keep in mind while planning large
>> > installations?
>> I am using 3.7.11 and only problem is slow rebuild time when a disk
>> fails. It takes 8 days to heal a 8TB disk.(This might be related with
>> my EC configuration 16+4)
>> 3.9+ versions has some improvements about this but I cannot test them
>> yet...
>>
>> On Thu, Jun 29, 2017 at 2:49 PM, jkiebzak <jkieb...@gmail.com> wrote:
>> > Thanks for the reply. We will mainly use this for archival - near-cold
>> > storage.
>> >
>> >
>> > Anything, from your experience, to keep in mind while planning large
>> > installations?
>> >
>> >
>> > Sent from my Verizon, Samsung Galaxy smartphone
>> >
>> > ---- Original message ----
>> > From: Serkan Çoban <cobanser...@gmail.com>
>> > Date: 6/29/17 4:39 AM (GMT-05:00)
>> > To: Jason Kiebzak <jkieb...@gmail.com>
>> > Cc: Gluster Users <gluster-users@gluster.org>
>> > Subject: Re: [Gluster-users] Multi petabyte gluster
>> >
>> > I am currently using 10PB single volume without problems. 40PB is on
>> > the way. EC is working fine.
>> > You need to plan ahead with large installations like this. Do complete
>> > workload tests and make sure your use case is suitable for EC.
>> >
>> >
>> > On Wed, Jun 28, 2017 at 11:18 PM, Jason Kiebzak <jkieb...@gmail.com>
>> > wrote:
>> >> Has anyone scaled to a multi petabyte gluster setup? How well does
>> >> erasure
>> >> code do with such a large setup?
>> >>
>> >> Thanks
>> >>
>> >> ___
>> >> Gluster-users mailing list
>> >> Gluster-users@gluster.org
>> >> http://lists.gluster.org/mailman/listinfo/gluster-users
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Multi petabyte gluster

[Alastair Neil]

We are using 3.10 and have a 7 PB cluster.  We decided against 16+3 as the
rebuild time are bottlenecked by matrix operations which scale as the
square of the number of data stripes.  There are some savings because of
larger data chunks but we ended up using 8+3 and heal times are about half
compared to 16+3.

-Alastair

On 30 June 2017 at 02:22, Serkan Çoban <cobanser...@gmail.com> wrote:

> >Thanks for the reply. We will mainly use this for archival - near-cold
> storage.
> Archival usage is good for EC
>
> >Anything, from your experience, to keep in mind while planning large
> installations?
> I am using 3.7.11 and only problem is slow rebuild time when a disk
> fails. It takes 8 days to heal a 8TB disk.(This might be related with
> my EC configuration 16+4)
> 3.9+ versions has some improvements about this but I cannot test them
> yet...
>
> On Thu, Jun 29, 2017 at 2:49 PM, jkiebzak <jkieb...@gmail.com> wrote:
> > Thanks for the reply. We will mainly use this for archival - near-cold
> > storage.
> >
> >
> > Anything, from your experience, to keep in mind while planning large
> > installations?
> >
> >
> > Sent from my Verizon, Samsung Galaxy smartphone
> >
> >  Original message 
> > From: Serkan Çoban <cobanser...@gmail.com>
> > Date: 6/29/17 4:39 AM (GMT-05:00)
> > To: Jason Kiebzak <jkieb...@gmail.com>
> > Cc: Gluster Users <gluster-users@gluster.org>
> > Subject: Re: [Gluster-users] Multi petabyte gluster
> >
> > I am currently using 10PB single volume without problems. 40PB is on
> > the way. EC is working fine.
> > You need to plan ahead with large installations like this. Do complete
> > workload tests and make sure your use case is suitable for EC.
> >
> >
> > On Wed, Jun 28, 2017 at 11:18 PM, Jason Kiebzak <jkieb...@gmail.com>
> wrote:
> >> Has anyone scaled to a multi petabyte gluster setup? How well does
> erasure
> >> code do with such a large setup?
> >>
> >> Thanks
> >>
> >> ___
> >> Gluster-users mailing list
> >> Gluster-users@gluster.org
> >> http://lists.gluster.org/mailman/listinfo/gluster-users
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Multi petabyte gluster

[Serkan Çoban]

>Thanks for the reply. We will mainly use this for archival - near-cold storage.
Archival usage is good for EC

>Anything, from your experience, to keep in mind while planning large 
>installations?
I am using 3.7.11 and only problem is slow rebuild time when a disk
fails. It takes 8 days to heal a 8TB disk.(This might be related with
my EC configuration 16+4)
3.9+ versions has some improvements about this but I cannot test them yet...

On Thu, Jun 29, 2017 at 2:49 PM, jkiebzak <jkieb...@gmail.com> wrote:
> Thanks for the reply. We will mainly use this for archival - near-cold
> storage.
>
>
> Anything, from your experience, to keep in mind while planning large
> installations?
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
>
>  Original message 
> From: Serkan Çoban <cobanser...@gmail.com>
> Date: 6/29/17 4:39 AM (GMT-05:00)
> To: Jason Kiebzak <jkieb...@gmail.com>
> Cc: Gluster Users <gluster-users@gluster.org>
> Subject: Re: [Gluster-users] Multi petabyte gluster
>
> I am currently using 10PB single volume without problems. 40PB is on
> the way. EC is working fine.
> You need to plan ahead with large installations like this. Do complete
> workload tests and make sure your use case is suitable for EC.
>
>
> On Wed, Jun 28, 2017 at 11:18 PM, Jason Kiebzak <jkieb...@gmail.com> wrote:
>> Has anyone scaled to a multi petabyte gluster setup? How well does erasure
>> code do with such a large setup?
>>
>> Thanks
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Multi petabyte gluster

[jkiebzak]

Thanks for the reply. We will mainly use this for archival - near-cold storage.

Anything, from your experience, to keep in mind while planning large 
installations?

Sent from my Verizon, Samsung Galaxy smartphone
 Original message From: Serkan Çoban <cobanser...@gmail.com> 
Date: 6/29/17  4:39 AM  (GMT-05:00) To: Jason Kiebzak <jkieb...@gmail.com> Cc: 
Gluster Users <gluster-users@gluster.org> Subject: Re: [Gluster-users] Multi 
petabyte gluster 
I am currently using 10PB single volume without problems. 40PB is on
the way. EC is working fine.
You need to plan ahead with large installations like this. Do complete
workload tests and make sure your use case is suitable for EC.


On Wed, Jun 28, 2017 at 11:18 PM, Jason Kiebzak <jkieb...@gmail.com> wrote:
> Has anyone scaled to a multi petabyte gluster setup? How well does erasure
> code do with such a large setup?
>
> Thanks
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Multi petabyte gluster

[Serkan Çoban]

I am currently using 10PB single volume without problems. 40PB is on
the way. EC is working fine.
You need to plan ahead with large installations like this. Do complete
workload tests and make sure your use case is suitable for EC.


On Wed, Jun 28, 2017 at 11:18 PM, Jason Kiebzak  wrote:
> Has anyone scaled to a multi petabyte gluster setup? How well does erasure
> code do with such a large setup?
>
> Thanks
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Multi petabyte gluster

[Jason Kiebzak]

Has anyone scaled to a multi petabyte gluster setup? How well does erasure
code do with such a large setup?

Thanks
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users