Hi, if you want to compute local parities within a plug-in library by adding rows to the encoding matrix, you can still do that. If you can do LRC inside the plug-in more efficienty, one should do it in the plug-in and skip the LRC layer. I have a patch to the ISA plug-in which can do LRC like HDFS Xorbas encoding, but i didn't manage to express that in Loic's generic LRC layer ... I think it is hard to formulate a generic framework to define LRC as a layer on top and doing the encoding/decoding only in the lowest layer. Same problem for the shingled EC, which you cannot express in the simple LRC way ...
I see the current LRC layering as a generic tool to use in multi-datacenter setups honouring the locality of chunks because some of the LRC optimization codes break locality by cross-talk between locations, they reduce the traffic but do not restrict traffic within data centers. Cheers Andreas. ________________________________________ From: Loic Dachary [[email protected]] Sent: 17 November 2014 02:04 To: Zhou, Yuan; Andreas Joachim Peters Cc: [email protected] Subject: Re: Question on Ceph LRC design Hi, I believe Andreas has a more elaborate answer on this topic. The current implementation is not as good as what is described in the paper you mention, this is correct. My incentive for chosing this path is that I was able to understand it, mainly. It is not much more than stacking layers of erasure coded chunks on top of each other ;-) Now that we have this plugin, it would be nice to have another implementation that uses less space and possible even less network when reconstructing. During the OpenStack summit we discussed this with Kevin Greenan and there are promising directions. It would help a lot to have a sample code to study so that it can be adapted to what we have currently in Ceph. Do you know of such an implementation of LRC or other similar code designed to reduce the network bandwidth during reconstruction ? Cheers On 17/11/2014 01:52, Zhou, Yuan wrote: > Hi Loic/Anderas, > > > > I was trying to understand the LRC design in Ceph EC. Per my understanding, > it seems Ceph was using a slightly different design with the Microsoft LRC: > the local parities were calculated with the global parities included. Is > there any special consideration on this change? > > I was asking because in a typical MS LRC design the global and local parities > could be calculated at the same time actually(I mean inside the Erasure Code > library). But with this new design, we lost this potential optimization. > > > > Thanks, -Yuan > -- Loïc Dachary, Artisan Logiciel Libre -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
