Woah. That looks promising Jason. Can you hack up TestMemStore.java and have it see if above will even pass the CSLM hbase tests?
St.Ack On Tue, Jun 14, 2011 at 5:23 PM, Jason Rutherglen <[email protected]> wrote: > Here's the implementation we can presumably drop in and benchmark: > > https://github.com/mspiegel/lockfreeskiptree > > On Tue, Jun 14, 2011 at 4:56 PM, Jason Rutherglen > <[email protected]> wrote: >> Here's a paper on a ConcurrentSkipTree: >> >> http://www.cs.virginia.edu/~ms6ep/publications/michael-spiegel-dissertation.pdf >> >> On Tue, Jun 14, 2011 at 4:40 PM, Jason Rutherglen >> <[email protected]> wrote: >>> Right, I think one naive approach would be to take CSLM, convert it to >>> use AtomicIntegerArray instead of objects (basically emulate objects >>> in blocks of ints), and of course not implement delete functionality. >>> >>> The first challenge is making an AtomicIntegerArray based linked list >>> where the value is pointed to by an int into a byte block data >>> structure. The next is a 2nd AIA that emulates the internal 'Index' >>> class of CSLM. >>> >>> Thoughts? >>> >>> On Tue, Jun 14, 2011 at 3:51 PM, Stack <[email protected]> wrote: >>>> Or do as Todd suggested a while back and pull in CSLM and hack it to >>>> suit our needs; its public domain. >>>> St.Ack >>>> >>>> On Tue, Jun 14, 2011 at 3:23 PM, Jason Rutherglen >>>> <[email protected]> wrote: >>>>>> See this J: >>>>>> http://www.cloudera.com/blog/2011/03/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-3/ >>>>> >>>>> Interesting. >>>>> >>>>> In Lucene for realtime search we have the same problem where the >>>>> best/only option right now for the realtime terms dictionary is to use >>>>> ConcurrentSkipListMap. However the Node, key, value, and Index >>>>> objects all are pointers that consume RAM and need to be cleaned up. >>>>> >>>>> Lucene right now during indexing does not generate much object pointer >>>>> garbage. We have the same problem in the two projects. >>>>> >>>>> Maybe there's a way to create an AtomicIntegerArray based system that >>>>> uses int pointers to perform the same function as a the concurrent >>>>> skip list. It would greatly reduce garbage generation. >>>>> >>>>> On Tue, Jun 14, 2011 at 10:14 AM, Stack <[email protected]> wrote: >>>>>> See this J: >>>>>> http://www.cloudera.com/blog/2011/03/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-3/ >>>>>> St.Ack >>>>>> >>>>>> On Mon, Jun 13, 2011 at 10:58 PM, Jason Rutherglen >>>>>> <[email protected]> wrote: >>>>>>> Is there an issue with garbage generated from using CSLM in the >>>>>>> MemStore? >>>>>>> >>>>>> >>>>> >>>> >>> >> >
