On Sun, 18 Aug 2013, Loic Dachary wrote:
> Hi Sage,
>
> Unless I misunderstood something ( which is still possible at this stage ;-)
> decode() is used both for recovery of missing chunks and retrieval of the
> original buffer. Decoding the M data chunks is a special case of decoding N
> <= M chunks out of the M+K chunks that were produced by encode(). It can be
> used to recover parity chunks as well as data chunks.
>
> https://github.com/dachary/ceph/blob/wip-4929/doc/dev/osd_internals/erasure-code.rst#erasure-code-library-abstract-api
>
> map<int, buffer> decode(const set<int> &want_to_read, const map<int,
> buffer> &chunks)
>
> decode chunks to read the content of the want_to_read chunks and return a
> map associating the chunk number with its decoded content. For instance, in
> the simplest case M=2,K=1 for an encoded payload of data A and B with parity
> Z, calling
>
> decode([1,2], { 1 => 'A', 2 => 'B', 3 => 'Z' })
> => { 1 => 'A', 2 => 'B' }
>
> If however, the chunk B is to be read but is missing it will be:
>
> decode([2], { 1 => 'A', 3 => 'Z' })
> => { 2 => 'B' }
Ah, I guess this works when some of the chunks contain the original
data (as with a parity code). There are codes that don't work that way,
although I suspect we won't use them.
Regardless, I wonder if we should generalize slightly and have some
methods work in terms of (offset,length) of the original stripe to
generalize that bit. Then we would have something like
map<int, buffer> transcode(const set<int> &want_to_read, const map<int,
buffer>& chunks);
to go from chunks -> chunks (as we would want to do with, say, a LRC-like
code where we can rebuild some shards from a subset of the other shards).
And then also have
int decode(const map<int, buffer>& chunks, unsigned offset,
unsigned len, bufferlist *out);
that recovers the original data.
In our case, the read path would use decode, and for recovery we would use
transcode.
We'd also want to have alternate minimum_to_decode* methods, like
virtual set<int> minimum_to_decode(unsigned offset, unsigned len, const
set<int> &available_chunks) = 0;
What do you think?
sage
>
> Cheers
>
> On 18/08/2013 19:34, Sage Weil wrote:
> > On Sun, 18 Aug 2013, Loic Dachary wrote:
> >> Hi Ceph,
> >>
> >> I've implemented a draft of the Erasure Code plugin loader in the context
> >> of http://tracker.ceph.com/issues/5878. It has a trivial unit test and an
> >> example plugin. It would be great if someone could do a quick review. The
> >> general idea is that the erasure code pool calls something like:
> >>
> >> ErasureCodePlugin::factory(&erasure_code, "example", parameters)
> >>
> >> as shown at
> >>
> >> https://github.com/ceph/ceph/blob/5a2b1d66ae17b78addc14fee68c73985412f3c8c/src/test/osd/TestErasureCode.cc#L28
> >>
> >> to get an object implementing the interface
> >>
> >> https://github.com/ceph/ceph/blob/5a2b1d66ae17b78addc14fee68c73985412f3c8c/src/osd/ErasureCodeInterface.h
> >>
> >> which matches the proposal described at
> >>
> >> https://github.com/dachary/ceph/blob/wip-4929/doc/dev/osd_internals/erasure-code.rst#erasure-code-library-abstract-api
> >>
> >> The draft is at
> >>
> >> https://github.com/ceph/ceph/commit/5a2b1d66ae17b78addc14fee68c73985412f3c8c
> >>
> >> Thanks in advance :-)
> >
> > I haven't been following this discussion too closely, but taking a look
> > now, the first 3 make sense, but
> >
> > virtual map<int, bufferptr> decode(const set<int> &want_to_read, const
> > map<int, bufferptr> &chunks) = 0;
> >
> > it seems like this one should be more like
> >
> > virtual int decode(const map<int, bufferptr> &chunks, bufferlist *out);
> >
> > As in, you'd decode the chunks you have to get the actual data. If you
> > want to get (missing) chunks for recovery, you'd do
> >
> > minimum_to_decode(...); // see what we need
> > <fetch those chunks from other nodes>
> > decode(...); // reconstruct original buffer
> > encode(...); // encode missing chunks from original data
> >
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
> --
> Lo?c Dachary, Artisan Logiciel Libre
> All that is necessary for the triumph of evil is that good people do nothing.
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
