On Tue, Dec 20, 2016 at 5:59 AM, Ravikumar Govindarajan < [email protected]> wrote:
> Thanks a lot Aaron. > > One quick clarification... Is it possible for you to provide an off-heap > option for the _pointers Map, since we already have Slab allocation and > related functionality in place? > I'm sure it's possible however the main use case didn't use a lot of memory so that sort of feature was never implemented. > > I guess the challenge is to use Unsafe buffers optimally, for varying > lengths of values. > Yeah that would be a problem if variable blocks are stored. Perhaps if you really need a WAL that you could look at the sync/fencing code in the KV store to implement a WAL. Also HBase uses (or did at one point) a standard sequence file for a WAL. > > -- > Ravi > > On Thu, Dec 15, 2016 at 7:17 PM, Aaron McCurry <[email protected]> wrote: > > > Take a look at: > > > > https://github.com/apache/incubator-blur/blob/master/blur-kvs > > > > https://github.com/apache/incubator-blur/blob/master/ > > blur-kvs/src/main/java/org/apache/blur/kvs/HdfsKeyValueStore.java > > > > https://github.com/apache/incubator-blur/blob/master/ > > blur-kvs/src/test/java/org/apache/blur/kvs/HdfsKeyValueStoreTest.java > > > > The HdfsKeyValueStore took the place of the WAL in Blur. It's an > inmemory > > cache that needs to be purged (and written to disk) from time to time. > > > > Let me know if you have questions. > > > > Aaron > > > > On Thu, Dec 15, 2016 at 7:22 AM, Ravikumar Govindarajan < > > [email protected]> wrote: > > > > > I remember seeing Blur writing transaction logs (WAL) in HDFS but is > > > removed as of today... > > > > > > Was re-implementing WAL-log for a particular use-case in Blur. Would be > > > great if community experienced any issues during WAL-writes in HDFS & > > could > > > share them here... > > > > > > I was looking at corner cases where WAL files could go corrupt. I guess > > > there should be many scenarios but just off the top of my head > > > > > > - Program crashing on OOM/Seg-fault, with open WAL streams to HDFS > > > - Name-Node, Data-Node differing on length of written block-data > > > - Data-Node(s) failing to ack sync calls correctly > > > > > > Any insights is much appreciated > > > > > > -- > > > Ravi > > > > > >
