I have created OAK-891 to track this. @Tommaso - Thanks for looking into this. I have couple of issues in current DirectMemory implementation. Would open JIRA and follow up there.
Chetan Mehrotra On Tue, Jul 9, 2013 at 3:15 PM, Tommaso Teofili <[email protected]> wrote: > > On 08/lug/2013, at 12:54, Chetan Mehrotra wrote: > >> Hi, >> >> I tried to integrate Apache DirectMemory as Second level cache (L2). >> It uses Kryo [2] for serializing/derserializing objects from the off >> heap memory managed by DirectMemory. Initial test based on this show >> quite a bit of saving in terms of memory taken >> >> Below stats (sum of all MongoMK related cache) are taken when L2 >> memory is enabled after restarting a CQ instance and some basic access >> done. >> >> On Heap >> No of entries - 138288 >> Memory Taken - 217.3 MB >> Off Heap >> No of entries - 113012 >> Memory Taken - 42.8 MB >> Hit Count 1257 >> Total Load Time 80253030 >> Avg Load Time 63844.8926014320 >> >> If I completely serialize the Document cache (the biggest of 3 ) >> - Document Cache - On Heap 134.7 MB for 43935 entries >> - Document Cache - Off Heap 27.5 MB for 43935 entries. Takes 555.5 ms >> to deserialize all 43935 entries >> >> So far I can make following observation >> -- Storing cache objects in serialized form provides a more compact storage >> -- Deserializtion cost is low >> -- Using Off heap memory would be helpful compared to on heap memory >> -- Even if we do not use DirectMemory we can look into storing the >> data in serialized form >> -- L2 cache can be added as an optional feature. Oak Core can still be >> used in absence of L2 cache >> >> Caveats >> -- DirectMemory project is still in very early phase and I have to use >> the latest snapshot version with some local fixes to get it work >> >> The required changes to be done can be checked at [3] >> >> So would it be worthwhile to add this as an optional feature in Oak. >> If user require they can enable L2 cache otherwise we only use the >> Guava Cache as L1 cache >> >> Thoughts? > > I think it makes sense, probably as an optional feature with a pluggable > implementation (can be DM or some other mechanism). > Regarding DM I think I the community would be happy to support our use case > (and I'd be happy to help on that). > > Regards, > Tommaso > >> >> Chetan Mehrotra >> >> [1] http://directmemory.apache.org/ >> [2] https://code.google.com/p/kryo/ >> [3] >> https://github.com/chetanmeh/jackrabbit-oak/compare/apache:ab955ffd40e23a9f8aee87f9076bc045b643e35d...offheap >
