The Netty off-heap allocator (or rather a Java port of jemalloc) is exactly the sort of thing I had in mind.
Tristan On 01/15/2014 12:59 PM, Jaromir Hamala wrote: > Hi, > > another option is to use the off-heap allocator from Netty project. > It's just an allocator, so it would require extra work when compared > with HugeCollections. > I tried this approach approach with Hazelcast: > https://github.com/jerrinot/hugecast > > Cheers, > Jaromir > > > On Wed, Jan 15, 2014 at 11:44 AM, Tristan Tarrant <[email protected] > <mailto:[email protected]>> wrote: > > Hi Ben, > > HugeCollections does indeed look interesting, and we'd gladly accept a > DataContainer implementation as you propose :) > > Before you start working on it, however, I'd better expand on the > reasons for implementing an off-heap DataContainer: we would > ultimately > like to have cache entries directly accessible as (Direct)ByteBuffers > which means we could then use NIO2 to directly transfer that data over > the network (JGroups, HotRod, etc) without additional copy operations. > HC uses Unsafe.allocateMemory() which does not have this facility, > so we > would still have to copy data from there to a heap-based ByteBuffer > before being able to pass that on to any NIO2 methods. > > This is all in the initial planning stages, so any comments are > welcome > > Tristan > > On 01/15/2014 11:54 AM, cotton-ben wrote: > > Hi Yavuz, Tristan, et. al. > > > > I am extremely interested to learn if anything materialized from the > > https://issues.jboss.org/browse/ISPN-871 effort. > > > > If nothing materialized, I would like to take a stab at doing this, > > specifically by doing the following: > > > > > > 0. Use Peter Lawrey's openHFT HugeCollections project > > (https://github.com/OpenHFT/HugeCollections) as the off-Heap Cache > > implementation provider. > > > > 1. Start with Peter's net.openhft.collections.HugeHashMap<K,V> > > implementation Class as a highly optimized off-Heap basis and a > potential > > Cache<K,V> candidate > > > > 2. Confirm from ISPN-dev team that the ambition to use > > org.infinispan.container.DataContainer interface as a bridge to > provide > > potential non-ISPN built CacheImpl<K,V> candidates is > sound/complete (and > > intended) > > > > 3. Modify HugheHashMap<K,V> so that it explicitly implements > > org.infinispan.container.DataContainer interface. > > > > 4. Confirm that modified my > net.openhft.collections..HugeHashMap<K,V> can > > interoperate with the full ISPN 5.3/6.x APIs, exactly as if it > were a > > default ISPN-provided org.infinispan.CacheImpl<K,V> > > > > Could any one from the ISPN-dev team comment if this ambition > has merit and > > a liklihood of "working" as outlined above (effectively resuming > the work > > started at https://issues.jboss.org/browse/ISPN-871)? Is there > any in-place > > ISPN documentation that advocates the use of DataContainer for > taking on > > this type of effort? > > > > Thanks, > > Ben > > > > > > > > -- > > View this message in context: > > http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-Infinispan-embedded-off-heap-cache-tp4026102p4028642.html > > Sent from the Infinispan Developer List mailing list archive at > Nabble.com. > > _______________________________________________ > > infinispan-dev mailing list > > [email protected] > <mailto:[email protected]> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > > _______________________________________________ > infinispan-dev mailing list > [email protected] <mailto:[email protected]> > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > -- > “Perfection is achieved, not when there is nothing more to add, but > when there is nothing left to take away.” > Antoine de Saint Exupéry > > > _______________________________________________ > infinispan-dev mailing list > [email protected] > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list [email protected] https://lists.jboss.org/mailman/listinfo/infinispan-dev
