Hi Denis, Yes we don't need to have expiration policy, so setting CacheConfiguraiton.setEagerTtl to false solved this problem.
So we are testing the whole system, we also found out that using a cache with writeBehind enabled causes a new thread creation for each one. So if we think about future plans and possibilities, it's not a best practice to have increasing number of threads within jvm. We wonder that if there is a option to use a thread pool for writeBehind jobs. If there is not, we could implement it for the community. So if you can guide us where to start, i would be glad :) Thanks. On Fri, Apr 8, 2016 at 1:54 AM, Denis Magda <[email protected]> wrote: > Tolga, > > The cleanup threads "ttl-cleanup-workers" are used to eagerly remove > expired cache entries. Expiration policy can be set either a cache wide in > CacheConfiguration or can be used later with cache.withExpirePolicy(...) > calls. > I failed to reproduce your case. What I've done is started 30 caches and > destroyed all of them later. Visual VM showed that all > "ttl-cleanup-workers" were stopped successfully. > What Ignite version do you use? > > In any case if you are not planing to use expiration policy you can set > CacheConfiguraiton.setEagerTtl to false and the ttl workers Threads won't > be created at all. > > Regards, > Denis > > > On 4/7/2016 3:43 PM, Tolga Kavukcu wrote: > > Hi Denis, > > IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE parameter seems like decreased > heap usage. I will run longer tests to check heap behaviour. > > Also i need another help with thread's created by ignite. I found out that > ignite creates a cleanup thread named "ttl-cleanup-worker" for each cache. > But when cache is destroyed, clean up thread does not deleted. Instead it > waits sleeping state at all. > > My first question is that , is it possible to decrease thread count with a > configuration, like "thread pool with x threads" for all caches. > Secondly, is "unremoved threads" are expected behaviour. > > Thanks. > > On Thu, Apr 7, 2016 at 2:40 PM, Denis Magda <[email protected]> wrote: > >> Hi Tolga, >> >> GridDhtPartitionTopologyImpl is created per cache. If you destroy a cache >> this object should be GCed. However you should use cache.destroy() for that. >> >> Please also make sure that you make "live set" heap dumps only. Try to >> perform GC explicitly before making the dump because a collector may clean >> dead objects much later depending on its heuristics. >> >> -- >> Denis >> >> On 4/7/2016 8:27 AM, Tolga Kavukcu wrote: >> >> Hi Denis, >> >> Thanks for the response. I will try IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE >> parameter. >> The screnshots taken from eclipse memory analyser which opens and analyses >> heap dump. I understand heap requirement for wrapping and indexing off-heap >> entry positions. But also found out that instances of >> *org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl >> *is constantly increasing within jvm. >> >> >> I also create and destroy so many small caches during the lifecycle, do >> you think that it is possible to destroyed caches leaves a footprint in >> heap. >> >> The previous scrreenshots was dominator tree view of memory analyser. I >> attached again with headers. >> >> You can see that each of GridDhtPartitionTopologyImpl uses 20mb~ heap. >> And there are 72 instances of GridDhtPartitionTopologyImpl living. >> >> Also i attached screenshots of leak suspect report of memory analyzer, >> which is taken periodically. You an see that instances of >> *GridDhtPartitionTopologyImpl >> keeps increasing. * >> >> Any ideas or suggestions? >> >> On Wed, Apr 6, 2016 at 6:00 PM, Denis Magda < <[email protected]> >> [email protected]> wrote: >> >>> Hi Tolga, >>> >>> GridDhtPartitionTopologyImpl contains list of partitions that belong to >>> a specific node. In case of offheap caches each partition (concurrent map) >>> contains set of wrappers around keys->values, stored offheap. The wrapper >>> holds information that's needed to unswap a value or a key to Java heap >>> from offheap when required by a user application. >>> So Ignite requires extra space for internal needs even when offheap mode >>> is used. >>> >>> I would recommend you trying to reduce >>> IgniteSystemProperties.IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE. This is the >>> size of the queue that keeps deleted entries for internal needs as well. >>> https://apacheignite.readme.io/v1.5/docs/capacity-planning >>> >>> BTW, could you explain what columns from your screenshot mean exactly? >>> What tool did you use to create the memory snapshot? >>> >>> -- >>> Denis >>> >>> >>> >>> On 4/6/2016 3:02 PM, Tolga Kavukcu wrote: >>> >>> Hi everyone, >>> >>> I use partitioned ignite cache for a very dynamic data. Means that there >>> are many updates,deletes and puts with around 5m~ row. >>> >>> So to avoid gc pauses i use off-heap mode. But when i analyse heap i see >>> that count and heap size of >>> *org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl* >>> is >>> increasing constantly. >>> >>> Please see attached screenshots taken from mat heap dump. >>> >>> <bean class="org.apache.ignite.configuration.CacheConfiguration" >>> name="DEFAULT"> <property name="atomicityMode" value="ATOMIC" /> >>> <property name="cacheMode" value="PARTITIONED" /> <property >>> name="memoryMode" value="OFFHEAP_TIERED" /> <property name="backups" >>> value="1" /> <property name="affinity"> <bean >>> class="org.apache.ignite.cache.affinity.fair.FairAffinityFunction"> >>> <constructor-arg index="0" type="int" value="6"/> </bean> >>> </property> <property name="writeThrough" value="false" /> <property >>> name="writeBehindEnabled" value="false" /></bean> >>> >>> Thanks for helping out. >>> >>> There are totally 1.2 heap used by GridDhtPartitionTopologyImpl, almost >>> equals to my data size. Do you think that there are problems with >>> configuration. >>> >>> >>> *Tolga KAVUKÇU * >>> >>> >>> >> >> >> -- >> >> *Tolga KAVUKÇU * >> >> >> > > > -- > > *Tolga KAVUKÇU * > > > -- *Tolga KAVUKÇU*
