Hi Isaeed Mohanna, I don't see any eviction or expired policy configured. Is entry deletion performed by you application?
Have you try to detect which of caches id grows unexpectedly? Have you analyse GC logs or tried to tune GC? Actually, you can putting data faster as garbage is collecting. This page may be helpful http://apacheignite.gridgain.org/v1.7/docs/performance-tips#tune-garbage-collection . Also you can get profile (with e.g. JavaFlightRecorder) of grid under load to understand what is really going on. Please let me know, if there are any issues. On Thu, Nov 10, 2016 at 10:10 AM, Isaeed Mohanna <[email protected]> wrote: > Hi > My cache configurations appear below. > > // Cache 1 - a cache of ~15 entities that has a date stamp that is updated > every 30 - 120 seconds > CacheConfiguration<?, ?> Cache1Cfg = new CacheConfiguration<>(); > Cache1Cfg cheCfg.setName("Cache1Name"); > Cache1Cfg .setCacheMode(CacheMode.REPLICATED); > Cache1Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC); > Cache1Cfg .setStartSize(50); > > // Cache 2 - A cache used as an ignite queue with frequent inserts and > removal from the queue > CacheConfiguration<?, ?> Cache2Cfg = new CacheConfiguration<>(); > Cache2Cfg .setName("Cache2Name"); > Cache2Cfg .setCacheMode(CacheMode.REPLICATED); > Cache2Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC); > > // Cache 3 - hundreds of entities updated daily > CacheConfiguration<?, ?> Cache3Cfg = new CacheConfiguration<>(); > Cache3Cfg .setName("Cache3Name"); > Cache3Cfg .setCacheMode(CacheMode.REPLICATED); > Cache3Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC); > Cache3Cfg .setIndexedTypes(UUID.class, SomeClass.class); > > // Cache 4 - Cache with very few writes and reads > CacheConfiguration<?, ?> Cache4Cfg = new CacheConfiguration<>(); > Cache4Cfg .setName("Cache4Name"); > Cache4Cfg .setCacheMode(CacheMode.REPLICATED); > Cache4Cfg .setAtomicityMode(CacheAtomicityMode.ATOMIC); > > // Events Cache - cache with very frequent writes and delete, acts as > events queue > CacheConfiguration<?, ?> eventsCacheConfig= new CacheConfiguration<>(); > eventsCacheConfig.setName("EventsCache"); > eventsCacheConfig.setCacheMode(CacheMode.PARTITIONED); > eventsCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC); > eventsCacheConfig.setIndexedTypes(UUID.class, SomeClass.class); > eventsCacheConfig.setBackups(1); > eventsCacheConfig.setOffHeapMaxMemory(0); > > // Failed Events Cache - cache with less writes and reads stores failed > events > CacheConfiguration<?, ?> failedEventsCacheConfig = new > CacheConfiguration<>(); > failedEventsCacheConfig.setName("FailedEventsCache"); > failedEventsCacheConfig.setCacheMode(CacheMode.PARTITIONED); > failedEventsCacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC); > failedEventsCacheConfig.setIndexedTypes(UUID.class, EventEntity.class); > failedEventsCacheConfig.setBackups(1); > failedEventsCacheConfig.setOffHeapMaxMemory(0); > > // In addition i have one atomic reference > AtomicConfiguration atomicCfg = new AtomicConfiguration(); > atomicCfg.setCacheMode(CacheMode.REPLICATED); > Thanks again > > On Wed, Nov 9, 2016 at 5:26 PM, Andrey Mashenkov <[email protected]> > wrote: > >> Hi Isaeed Mohanna, >> >> Would you please provide your cache configurations? >> >> >> On Wed, Nov 9, 2016 at 5:37 PM, Isaeed Mohanna <[email protected]> wrote: >> >>> Hi >>> i have an ignite 1.7.0 cluster with 3 nodes running , i have 3 >>> PARTITIONED >>> ATOMIC CACHES and 2 REPLICATED ATOMIC CACHES, Most of these caches are >>> populated with events data, so each cache entry is short lived its >>> inserted, >>> processed later by some task and removed. so the caches are pretty much >>> very >>> dynamic. >>> Recently the load in our system has increased (more events were received >>> and >>> generated) and we started experiencing out of memory fails once in while >>> (several days depending on machine size). >>> I have created several heap dumps and noticed the largest retained >>> objects >>> in memory is by the following classes: GridDhtLocalPartition, >>> ConccurentHashMap8,ConccurentHashMap8$Node[]. >>> I can see the GridDhtLocalPartition has a ConccurentHashMap8 so most >>> likely >>> all three reference the same thing. >>> My question what is this class and why does it retain memory, entities >>> in my >>> caches are usually short lived (several minutes in most caches) so i >>> would >>> expect the memory to be released? any hints on how to continue my >>> investigation would be great. >>> Thanks >>> >>> >>> >>> >>> >>> -- >>> View this message in context: http://apache-ignite-users.705 >>> 18.x6.nabble.com/Cache-Memory-Behavior-GridDhtLocalPartition-tp8835.html >>> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >>> >> >> >
