Re: [ceph-users] Need help to develop CEPH EC Plugin for array type of Erasure Code
Thanks for the follow-up. Lets start with normal erasure coding plugin. Here we have k data node and m parity nodes. For RDP or RAID-DP or MSR/MBR kind of erasure code the codeword is two dimensional array. FOr example, in case of RDP Erasure code the cored is kx(k+m) array. The codeword has k rows and n columns (i.e. k + m) The first k columns represents the systematic data and the last m columns represents the parity data. Here the object to be divided into kX(k+m) symbols. The RDP Erasure code has one row parity and rests are diagonal parity. Pl. make google image for RDP Erasure Code diagram. I'm I making sense... Thanks, Syed On Mon, Feb 1, 2016 at 9:03 PM, Syed Hussain wrote: > Hi, > > I've been working to develop a CEPH EC plgin for array type erasure code, > for example > RAID-DP (i.e RDP). Later I realized that I can't continue with (k+m) > format in CEPH as > like normal RS code if I want to save disk IO (or Read data). Since I > need to ready the individual symbols from each device rather that k > symbols. One "chunk" from OSD will contain k-symbols > in contiguous form. The same thing will be an issue for MSR(Minimum > Storage Regeneration) Erasure code plugin. > > As a solution I thought to consider k*k data chunk and k*m coding chunk > and use LRC kind of > layering to pass the info (in plugin) which is data chunk and coding chunk. > However, I've doubt if the decoding API will supply me the correct info on > lost chunks to get the info on lost erasure coding nodes. > > Does anybody have good solution to this issue using CRUSH map? > Could Pl. suggest any pointer or link that explain the above issue? > > Thanks, > Syed Abid Hussain > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Need help to develop CEPH EC Plugin for array type of Erasure Code
On Mon, Feb 1, 2016 at 7:33 AM, Syed Hussain wrote: > Hi, > > I've been working to develop a CEPH EC plgin for array type erasure code, > for example > RAID-DP (i.e RDP). Later I realized that I can't continue with (k+m) format > in CEPH as > like normal RS code if I want to save disk IO (or Read data). Since I need > to ready the individual symbols from each device rather that k symbols. One > "chunk" from OSD will contain k-symbols > in contiguous form. The same thing will be an issue for MSR(Minimum Storage > Regeneration) Erasure code plugin. > > As a solution I thought to consider k*k data chunk and k*m coding chunk and > use LRC kind of > layering to pass the info (in plugin) which is data chunk and coding chunk. > However, I've doubt if the decoding API will supply me the correct info on > lost chunks to get the info on lost erasure coding nodes. > > Does anybody have good solution to this issue using CRUSH map? > Could Pl. suggest any pointer or link that explain the above issue? I'm not really familiar with some of the terminology you're using and I suspect that's the case for most people on the list. Perhaps you could expand on the exact requirements for the erasure code you're trying to implement, and the issues with doing that through the current EC plugin interface? -Greg ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Need help to develop CEPH EC Plugin for array type of Erasure Code
Hi, I've been working to develop a CEPH EC plgin for array type erasure code, for example RAID-DP (i.e RDP). Later I realized that I can't continue with (k+m) format in CEPH as like normal RS code if I want to save disk IO (or Read data). Since I need to ready the individual symbols from each device rather that k symbols. One "chunk" from OSD will contain k-symbols in contiguous form. The same thing will be an issue for MSR(Minimum Storage Regeneration) Erasure code plugin. As a solution I thought to consider k*k data chunk and k*m coding chunk and use LRC kind of layering to pass the info (in plugin) which is data chunk and coding chunk. However, I've doubt if the decoding API will supply me the correct info on lost chunks to get the info on lost erasure coding nodes. Does anybody have good solution to this issue using CRUSH map? Could Pl. suggest any pointer or link that explain the above issue? Thanks, Syed Abid Hussain ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com