Hi There is no eviction policy since entries in the caches are removed by my application. (calling IgniteCache.remove ).
Digging through the core dump I can see that most resident items are cache entries were the cachecontext.cachename points to EventsCache, as i have mentioned before this cache has very frequent writes and deletions of events (i am using remove to delete the events), however this cache is also atomic,partitioned and have a backup of at least one so in case a node fails the event is not lost. when calling remove on a cache, is the backup of an entry removed as well? is it possible that the backup is not being cleaned? Currently i am using the default garbage collector settings, i can't see any spikes in performance due to GC, since i experience memory outage in several days i am not sure i am collecting data more than the GC is able to claim, I will try manually performing a GC when the system is about to crash to see wither forcing a GC will clean the memory. Thank you for ur help On Fri, Nov 11, 2016 at 11:59 AM, Andrey Mashenkov <[email protected]> wrote: > Hi Isaeed Mohanna, > > I don't see any eviction or expired policy configured. Is entry deletion > performed by you application? > > Have you try to detect which of caches id grows unexpectedly? > Have you analyse GC logs or tried to tune GC? Actually, you can putting > data faster as garbage is collecting. This page may be helpful > http://apacheignite.gridgain.org/v1.7/docs/performance-tips#tune-garbage- > collection. > > Also you can get profile (with e.g. JavaFlightRecorder) of grid under load > to understand what is really going on. > > Please let me know, if there are any issues. > > > > On Thu, Nov 10, 2016 at 10:10 AM, Isaeed Mohanna <[email protected]> > wrote: > >> Hi >> My cache configurations appear below. >> >> // Cache 1 - a cache of ~15 entities that has a date stamp that is >> updated every 30 - 120 seconds >> CacheConfiguration<?, ?> Cache1Cfg = new CacheConfiguration<>(); >> Cache1Cfg cheCfg.setName("Cache1Name"); >> Cache1Cfg .setCacheMode(CacheMode.REPLICATED); >> Cache1Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC); >> Cache1Cfg .setStartSize(50); >> >> // Cache 2 - A cache used as an ignite queue with frequent inserts and >> removal from the queue >> CacheConfiguration<?, ?> Cache2Cfg = new CacheConfiguration<>(); >> Cache2Cfg .setName("Cache2Name"); >> Cache2Cfg .setCacheMode(CacheMode.REPLICATED); >> Cache2Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC); >> >> // Cache 3 - hundreds of entities updated daily >> CacheConfiguration<?, ?> Cache3Cfg = new CacheConfiguration<>(); >> Cache3Cfg .setName("Cache3Name"); >> Cache3Cfg .setCacheMode(CacheMode.REPLICATED); >> Cache3Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC); >> Cache3Cfg .setIndexedTypes(UUID.class, SomeClass.class); >> >> // Cache 4 - Cache with very few writes and reads >> CacheConfiguration<?, ?> Cache4Cfg = new CacheConfiguration<>(); >> Cache4Cfg .setName("Cache4Name"); >> Cache4Cfg .setCacheMode(CacheMode.REPLICATED); >> Cache4Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC); >> >> // Events Cache - cache with very frequent writes and delete, acts as >> events queue >> CacheConfiguration<?, ?> eventsCacheConfig= new CacheConfiguration<>(); >> eventsCacheConfig.setName("EventsCache"); >> eventsCacheConfig.setCacheMode(CacheMode.PARTITIONED); >> eventsCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC); >> eventsCacheConfig.setIndexedTypes(UUID.class, SomeClass.class); >> eventsCacheConfig.setBackups(1); >> eventsCacheConfig.setOffHeapMaxMemory(0); >> >> // Failed Events Cache - cache with less writes and reads stores failed >> events >> CacheConfiguration<?, ?> failedEventsCacheConfig = new >> CacheConfiguration<>(); >> failedEventsCacheConfig.setName("FailedEventsCache"); >> failedEventsCacheConfig.setCacheMode(CacheMode.PARTITIONED); >> failedEventsCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC); >> failedEventsCacheConfig.setIndexedTypes(UUID.class, EventEntity.class); >> failedEventsCacheConfig.setBackups(1); >> failedEventsCacheConfig.setOffHeapMaxMemory(0); >> >> // In addition i have one atomic reference >> AtomicConfiguration atomicCfg = new AtomicConfiguration(); >> atomicCfg.setCacheMode(CacheMode.REPLICATED); >> Thanks again >> >> On Wed, Nov 9, 2016 at 5:26 PM, Andrey Mashenkov <[email protected] >> > wrote: >> >>> Hi Isaeed Mohanna, >>> >>> Would you please provide your cache configurations? >>> >>> >>> On Wed, Nov 9, 2016 at 5:37 PM, Isaeed Mohanna <[email protected]> >>> wrote: >>> >>>> Hi >>>> i have an ignite 1.7.0 cluster with 3 nodes running , i have 3 >>>> PARTITIONED >>>> ATOMIC CACHES and 2 REPLICATED ATOMIC CACHES, Most of these caches are >>>> populated with events data, so each cache entry is short lived its >>>> inserted, >>>> processed later by some task and removed. so the caches are pretty much >>>> very >>>> dynamic. >>>> Recently the load in our system has increased (more events were >>>> received and >>>> generated) and we started experiencing out of memory fails once in while >>>> (several days depending on machine size). >>>> I have created several heap dumps and noticed the largest retained >>>> objects >>>> in memory is by the following classes: GridDhtLocalPartition, >>>> ConccurentHashMap8,ConccurentHashMap8$Node[]. >>>> I can see the GridDhtLocalPartition has a ConccurentHashMap8 so most >>>> likely >>>> all three reference the same thing. >>>> My question what is this class and why does it retain memory, entities >>>> in my >>>> caches are usually short lived (several minutes in most caches) so i >>>> would >>>> expect the memory to be released? any hints on how to continue my >>>> investigation would be great. >>>> Thanks >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: http://apache-ignite-users.705 >>>> 18.x6.nabble.com/Cache-Memory-Behavior-GridDhtLocalPartition >>>> -tp8835.html >>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >>>> >>> >>> >> >
