Re: erasure coding (sorry)

2013-04-23 Thread Christopher LILJENSTOLPE
Supposedly, on 2013-Apr-22, at 08.09 PDT(-0700), someone claiming to be Sage Weil scribed: On Mon, 22 Apr 2013, Christopher LILJENSTOLPE wrote: Supposedly, on 2013-Apr-22, at 01.10 PDT(-0700), someone claiming to be Loic Dachary scribed: Hi Christopher, You wrote A modified

Re: erasure coding (sorry)

2013-04-22 Thread Christopher LILJENSTOLPE
Supposedly, on 2013-Apr-18, at 14.31 PDT(-0700), someone claiming to be Plaetinck, Dieter scribed: On Thu, 18 Apr 2013 16:09:52 -0500 Mark Nelson mark.nel...@inktank.com wrote: @Bryan: I did come across cleversafe. all the articles around it seemed promising, but unfortunately it seems

Re: erasure coding (sorry)

2013-04-22 Thread Loic Dachary
Hi Christopher, You wrote A modified client/library could be used to store objects that should be sharded, vs standard ceph treatment. In this model, each shard would be written to a seperate PG, and each PG would we stored on exactly one OSD. but there is no way for a client to enforce the

Re: erasure coding (sorry)

2013-04-22 Thread Christopher LILJENSTOLPE
Supposedly, on 2013-Apr-22, at 01.10 PDT(-0700), someone claiming to be Loic Dachary scribed: Hi Christopher, You wrote A modified client/library could be used to store objects that should be sharded, vs standard ceph treatment. In this model, each shard would be written to a seperate

Re: erasure coding (sorry)

2013-04-22 Thread Sage Weil
On Mon, 22 Apr 2013, Christopher LILJENSTOLPE wrote: Supposedly, on 2013-Apr-22, at 01.10 PDT(-0700), someone claiming to be Loic Dachary scribed: Hi Christopher, You wrote A modified client/library could be used to store objects that should be sharded, vs standard ceph treatment.

Re: erasure coding (sorry)

2013-04-22 Thread Loic Dachary
Hi Sage, On 04/22/2013 05:09 PM, Sage Weil wrote: On Mon, 22 Apr 2013, Christopher LILJENSTOLPE wrote: Supposedly, on 2013-Apr-22, at 01.10 PDT(-0700), someone claiming to be Loic Dachary scribed: Hi Christopher, You wrote A modified client/library could be used to store objects that

Re: erasure coding (sorry)

2013-04-22 Thread Sage Weil
On Mon, 22 Apr 2013, Loic Dachary wrote: Hi Sage, On 04/22/2013 05:09 PM, Sage Weil wrote: On Mon, 22 Apr 2013, Christopher LILJENSTOLPE wrote: Supposedly, on 2013-Apr-22, at 01.10 PDT(-0700), someone claiming to be Loic Dachary scribed: Hi Christopher, You wrote A modified

Re: erasure coding (sorry)

2013-04-20 Thread Loic Dachary
Hi Christopher, I would like to offer my help on this blueprint. In http://wiki.ceph.com/01Planning/02Blueprints/Dumpling/Erasure_encoding_as_a_storage_backend you wrote At this time, Annai is more than willing to help with this, but we don't have the resources (including ceph coders) to

erasure coding (sorry)

2013-04-18 Thread Plaetinck, Dieter
sorry to bring this up again, googling revealed some people don't like the subject [anymore]. but I'm working on a new +- 3PB cluster for storage of immutable files. and it would be either all cold data, or mostly cold. 150MB avg filesize, max size 5GB (for now) For this use case, my impression

Re: erasure coding (sorry)

2013-04-18 Thread Sage Weil
On Thu, 18 Apr 2013, Plaetinck, Dieter wrote: sorry to bring this up again, googling revealed some people don't like the subject [anymore]. but I'm working on a new +- 3PB cluster for storage of immutable files. and it would be either all cold data, or mostly cold. 150MB avg filesize, max

Re: erasure coding (sorry)

2013-04-18 Thread Mark Nelson
On 04/18/2013 04:08 PM, Josh Durgin wrote: On 04/18/2013 01:47 PM, Sage Weil wrote: On Thu, 18 Apr 2013, Plaetinck, Dieter wrote: sorry to bring this up again, googling revealed some people don't like the subject [anymore]. but I'm working on a new +- 3PB cluster for storage of immutable

Re: erasure coding (sorry)

2013-04-18 Thread Noah Watkins
On Apr 18, 2013, at 2:08 PM, Josh Durgin josh.dur...@inktank.com wrote: I talked to some folks interested in doing a more limited form of this yesterday. They started a blueprint [1]. One of their ideas was to have erasure coding done by a separate process (or thread perhaps). It would use

Re: erasure coding (sorry)

2013-04-18 Thread Sage Weil
On Thu, 18 Apr 2013, Noah Watkins wrote: On Apr 18, 2013, at 2:08 PM, Josh Durgin josh.dur...@inktank.com wrote: I talked to some folks interested in doing a more limited form of this yesterday. They started a blueprint [1]. One of their ideas was to have erasure coding done by a separate

Re: erasure coding (sorry)

2013-04-18 Thread Christopher LILJENSTOLPE
Supposedly, on 2013-Apr-18, at 14.08 PDT(-0700), someone claiming to be Josh Durgin scribed: On 04/18/2013 01:47 PM, Sage Weil wrote: On Thu, 18 Apr 2013, Plaetinck, Dieter wrote: sorry to bring this up again, googling revealed some people don't like the subject [anymore]. but I'm working

Re: erasure coding (sorry)

2013-04-18 Thread Christopher LILJENSTOLPE
Supposedly, on 2013-Apr-18, at 14.31 PDT(-0700), someone claiming to be Plaetinck, Dieter scribed: On Thu, 18 Apr 2013 16:09:52 -0500 Mark Nelson mark.nel...@inktank.com wrote: On 04/18/2013 04:08 PM, Josh Durgin wrote: On 04/18/2013 01:47 PM, Sage Weil wrote: On Thu, 18 Apr 2013,

Re: erasure coding (sorry)

2013-04-18 Thread Christopher LILJENSTOLPE
Supposedly, on 2013-Apr-18, at 14.24 PDT(-0700), someone claiming to be Noah Watkins scribed: On Apr 18, 2013, at 2:08 PM, Josh Durgin josh.dur...@inktank.com wrote: I talked to some folks interested in doing a more limited form of this yesterday. They started a blueprint [1]. One of their

Re: erasure coding (sorry)

2013-04-18 Thread Christopher LILJENSTOLPE
Supposedly, on 2013-Apr-18, at 14.26 PDT(-0700), someone claiming to be Sage Weil scribed: On Thu, 18 Apr 2013, Noah Watkins wrote: On Apr 18, 2013, at 2:08 PM, Josh Durgin josh.dur...@inktank.com wrote: I talked to some folks interested in doing a more limited form of this yesterday. They