Thanks a lot Aaron. One quick clarification... Is it possible for you to provide an off-heap option for the _pointers Map, since we already have Slab allocation and related functionality in place?
I guess the challenge is to use Unsafe buffers optimally, for varying lengths of values. -- Ravi On Thu, Dec 15, 2016 at 7:17 PM, Aaron McCurry <[email protected]> wrote: > Take a look at: > > https://github.com/apache/incubator-blur/blob/master/blur-kvs > > https://github.com/apache/incubator-blur/blob/master/ > blur-kvs/src/main/java/org/apache/blur/kvs/HdfsKeyValueStore.java > > https://github.com/apache/incubator-blur/blob/master/ > blur-kvs/src/test/java/org/apache/blur/kvs/HdfsKeyValueStoreTest.java > > The HdfsKeyValueStore took the place of the WAL in Blur. It's an inmemory > cache that needs to be purged (and written to disk) from time to time. > > Let me know if you have questions. > > Aaron > > On Thu, Dec 15, 2016 at 7:22 AM, Ravikumar Govindarajan < > [email protected]> wrote: > > > I remember seeing Blur writing transaction logs (WAL) in HDFS but is > > removed as of today... > > > > Was re-implementing WAL-log for a particular use-case in Blur. Would be > > great if community experienced any issues during WAL-writes in HDFS & > could > > share them here... > > > > I was looking at corner cases where WAL files could go corrupt. I guess > > there should be many scenarios but just off the top of my head > > > > - Program crashing on OOM/Seg-fault, with open WAL streams to HDFS > > - Name-Node, Data-Node differing on length of written block-data > > - Data-Node(s) failing to ack sync calls correctly > > > > Any insights is much appreciated > > > > -- > > Ravi > > >
