Hi Alexey, I see. That sounds reasonable on that level (and is something different). What is your plan to limit the overhead/impact of detecting and moving around such small chunks?
Kind regards, Peter 2017-02-03 14:52 GMT+01:00 Alexey Goncharuk <[email protected]>: > Peter, > > A page is a minimal entity that is allowed to be moved on OS level. If I > have a chunk of memory allocated in the middle of a page, OS won't be able > to move it during defragmentation (at least this is how things worked last > time I checked). In the new approach, we are able to move this chunk within > the page, this is the key difference compared to the old off-heap approach. > > Note, that fine-grained per-entry locking does not imply a byte array per > entry. We will still have our old GridCacheMapEntry object, but it will be > created only during a cache read or update, and will be GC-ed immediately > after cache operation completes. A value itself, on the other hand, will be > stored off-heap. Your measurements confirm that pure off-heap cache > outperforms both options for on-heap caches in terms of GC, so I do not see > any problems here. > > Thanks, > Alexey > > 2017-01-26 23:23 GMT+03:00 Peter Schmitt <[email protected]>: > >> Hi Alexey, >> >> a modern OS won't give you a physical address. Instead you get virtual >> memory which allows the OS to support features like defragmentation. >> "fine-grained per-entry" sounds like a byte-array for every entry. >> We tested something like that and the result is again higher GC pauses. >> In our case a FGC with off-heap caches takes e.g. up to 1s. >> With the same data and using on-heap caches it takes up to 25s. >> With a byte-array per entry we get about 4s for a FGC. >> With some optimizations you might get down a bit, however, in view of the >> FGC time I fear that such an approach causes a significant impact for >> applications which need to keep huge caches. >> >> Kind regards, >> Peter >> >> >> >> 2017-01-26 15:57 GMT+01:00 Alexey Goncharuk <[email protected]>: >> >>> Hi Peter, >>> >>> Leaving defragmentation to Ignite is one of the reasons we are trying >>> PageMemory approach. In Ignite 1.x we basically use OS memory allocator to >>> place a value off-heap. Once the OS has given us a pointer, the memory >>> cannot be moved around unless we free this region, thus the fragmentation. >>> On the other hand, with PageMemory approach we can define the page layout >>> ourselves and, given fine-grained per-entry locking, move values around in >>> memory. This makes memory management more complicated and usually defines >>> some restrictions on the data structures, but we expect this will give us >>> more control over the data. >>> >>> For example, one of the tricks we can do in the future is cluster-wide >>> cache snapshots: we simply dump the continuous region of memory on disk and >>> get a node-local caches snapshot. This is impossible to do using current >>> off-heap approach. >>> >>> --AG >>> >>> 2017-01-25 12:32 GMT+03:00 Peter Schmitt <[email protected]>: >>> >>>> Hi Val, >>>> >>>> fair enough :-) >>>> However, in the source-code it looks like "Flexible and precise >>>> per-cache memory limit" is supported already. >>>> At least each GridUnsafeMemory instance (in each GridCacheContext) >>>> receives the value from CacheConfiguration.getOffHeapMaxMemory). >>>> I haven't debugged it, but the instances are created per cache. >>>> >>>> Furthermore, I'm not sure about the memory-page concept, because you >>>> would get pages on top of pages (used by a modern OS). >>>> Sounds like even more fragmentation and in this case Ignite would need >>>> to do the defragmentation. >>>> >>>> Kind regards, >>>> Peter >>>> >>>> >>>> >>>> 2017-01-25 1:56 GMT+01:00 vkulichenko <[email protected]>: >>>> >>>>> I don't think I will be able to explain better than it's done in the >>>>> ticket >>>>> :) Current approach is naive and not always effective for multiple >>>>> reasons. >>>>> >>>>> -Val >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: http://apache-ignite-users.705 >>>>> 18.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-tp1 >>>>> 0218p10235.html >>>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >>>>> >>>> >>>> >>> >> >
