[ https://issues.apache.org/jira/browse/IGNITE-7319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389681#comment-16389681 ]
Zbyszek B commented on IGNITE-7319: ----------------------------------- {noformat} C:\Dev>"C:\Program Files\Java\jdk1.8.0_111\bin\java" -Xmx3G "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2017.2.5\lib\idea_rt.jar=4917 1:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2017.2.5\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\Java\jdk1.8.0_111\jre\lib\charsets .jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_11 1\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\ext\jaccess.jar;C:\Program Files\ Java\jdk1.8.0_111\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\ext\nashorn.jar ;C:\Program Files\Java\jdk1.8.0_111\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.8.0_111\ jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\ext\zipfs.jar;C:\Program Files \Java\jdk1.8.0_111\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\jfr.jar;C:\Program Files\Jav a\jdk1.8.0_111\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\rt.jar;C:\Dev\Ig niteMemLeak2\out\production\IgniteDemo;C:\Dev\IgniteMemLeak2\lib\2.3\cache-api-1.0.0.jar;C:\Dev\IgniteMemLeak2\lib\2.3\commons-codec-1.6.jar;C:\Dev\IgniteMemLea k2\lib\2.3\h2-1.4.195.jar;C:\Dev\IgniteMemLeak2\lib\2.3\ignite-core-2.3.0.jar;C:\Dev\IgniteMemLeak2\lib\2.3\ignite-indexing-2.3.0.jar;C:\Dev\IgniteMemLeak2\lib\ 2.3\lucene-analyzers-common-5.5.2.jar;C:\Dev\IgniteMemLeak2\lib\2.3\lucene-core-5.5.2.jar;C:\Dev\IgniteMemLeak2\lib\2.3\lucene-queryparser-5.5.2.jar" p1.Demo >>> Starting Ignite on MyLeakingNode; work directory C:\Users\testuser\ignite >>> ... Mar 06, 2018 6:12:16 PM java.util.logging.LogManager$RootLogger log SEVERE: Failed to resolve default logging config file: config/java.util.logging.properties [18:12:16] __________ ________________ [18:12:16] / _/ ___/ |/ / _/_ __/ __/ [18:12:16] _/ // (7 7 // / / / / _/ [18:12:16] /___/\___/_/|_/___/ /_/ /___/ [18:12:16] [18:12:16] ver. 2.3.0#20171028-sha1:8add7fd5 [18:12:16] 2017 Copyright(C) Apache Software Foundation [18:12:16] [18:12:16] Ignite documentation: http://ignite.apache.org [18:12:16] [18:12:16] Quiet mode. [18:12:16] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat} [18:12:16] [18:12:16] OS: Windows 7 6.1 amd64 [18:12:16] VM information: Java(TM) SE Runtime Environment 1.8.0_111-b14 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.111-b14 [18:12:16] Initial heap size is 256MB (should be no less than 512MB, use -Xms512m -Xmx512m). [18:12:16] Configured plugins: [18:12:16] ^-- None [18:12:16] [18:12:16] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message qu eues growth on sender and receiver sides. [18:12:16] Security status [authentication=off, tls/ssl=off] [18:12:18] Performance suggestions for grid (fix if possible) [18:12:18] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true [18:12:18] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options) [18:12:18] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options) [18:12:18] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) [18:12:18] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [18:12:18] [18:12:18] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [18:12:18] [18:12:18] Ignite node started OK (id=8219875e) [18:12:18] Topology snapshot [ver=1, servers=1, clients=0, CPUs=8, heap=2.7GB] >>> Ignite started on MyLeakingNode (8219875e-97d5-4e0e-852a-15eb83759b7d) >>> successfully! s Queue size: 1 Caches created: 387307 Caches destroyed: 387303 Mar 07, 2018 4:20:56 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Error when executing timeout callback: CancelableTask [id=c4ad4810261-a07dab65-3c81-4349-af11-68b9d200e3b3, endTime=1520392859178, period=5000, cancel=f alse, task=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager$BackupCleaner@1b3cca44] java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:850) at java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1962) at org.apache.ignite.internal.util.GridConcurrentSkipListSet.add(GridConcurrentSkipListSet.java:143) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor.addTimeoutObject(GridTimeoutProcessor.java:95) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask.onTimeout(GridTimeoutProcessor.java:267) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:163) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) Mar 07, 2018 4:21:29 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Runtime error caught during grid runnable execution: GridWorker [name=grid-timeout-worker, igniteInstanceName=null, finished=false, hashCode=223614316, interrupted=false, runner=grid-timeout-worker-#23] java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:850) at java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1962) at org.apache.ignite.internal.util.GridConcurrentSkipListSet.add(GridConcurrentSkipListSet.java:143) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor.addTimeoutObject(GridTimeoutProcessor.java:95) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask.onTimeout(GridTimeoutProcessor.java:267) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:163) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) Exception in thread "grid-timeout-worker-#23" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:850) at java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1962) at org.apache.ignite.internal.util.GridConcurrentSkipListSet.add(GridConcurrentSkipListSet.java:143) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor.addTimeoutObject(GridTimeoutProcessor.java:95) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask.onTimeout(GridTimeoutProcessor.java:267) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:163) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) Mar 07, 2018 4:23:36 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Failed to unmarshal discovery custom message. java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.HashMap$KeySet.iterator(HashMap.java:916) at java.util.HashSet.iterator(HashSet.java:172) at java.util.AbstractCollection.addAll(AbstractCollection.java:343) at org.apache.ignite.internal.managers.discovery.DiscoCache.<init>(DiscoCache.java:123) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.createDiscoCache(GridDiscoveryManager.java:2280) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.access$1700(GridDiscoveryManager.java:160) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:712) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery(GridDiscoveryManager.java:571) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.notifyDiscoveryListener(ServerImpl.java:5463) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processCustomMessage(ServerImpl.java:5282) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2656) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2447) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6648) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2533) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) Mar 07, 2018 4:24:13 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Failed to initialize cache. Will try to rollback cache start routine. [cacheName=localCache11251307] class org.apache.ignite.IgniteCheckedException: Failed to register MBean for component: org.apache.ignite.internal.processors.cache.CacheClusterMetricsMXBeanImp l@204fbdb1 at org.apache.ignite.internal.processors.cache.GridCacheProcessor.registerMbean(GridCacheProcessor.java:3600) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCache(GridCacheProcessor.java:1566) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1799) at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:751) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(GridDhtPartitionsExchangeF uture.java:882) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:588) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2279) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:112) at org.apache.ignite.internal.util.IgniteUtils.makeCacheMBeanName(IgniteUtils.java:4484) at org.apache.ignite.internal.util.IgniteUtils.registerCacheMBean(IgniteUtils.java:4574) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.registerMbean(GridCacheProcessor.java:3596) ... 8 more Mar 07, 2018 9:51:20 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Runtime error caught during grid runnable execution: Socket reader [id=1708, name=tcp-disco-sock-reader-#8, nodeId=null] java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "tcp-disco-sock-reader-#8" java.lang.OutOfMemoryError: GC overhead limit exceeded Mar 07, 2018 9:51:30 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Runtime error caught during grid runnable execution: IgniteSpiThread [name=tcp-disco-ip-finder-cleaner-#5] java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "tcp-disco-ip-finder-cleaner-#5" java.lang.OutOfMemoryError: GC overhead limit exceeded Mar 07, 2018 9:51:49 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Runtime error caught during grid runnable execution: Socket reader [id=1706, name=tcp-disco-sock-reader-#6, nodeId=null] java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "tcp-disco-sock-reader-#6" java.lang.OutOfMemoryError: GC overhead limit exceeded Mar 07, 2018 9:52:05 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Runtime error caught during grid runnable execution: Socket reader [id=1707, name=tcp-disco-sock-reader-#7, nodeId=null] java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "tcp-disco-sock-reader-#7" java.lang.OutOfMemoryError: GC overhead limit exceeded Mar 07, 2018 9:52:37 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Runtime error caught during grid runnable execution: Socket reader [id=1709, name=tcp-disco-sock-reader-#9, nodeId=null] java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "tcp-disco-sock-reader-#9" java.lang.OutOfMemoryError: GC overhead limit exceeded Mar 07, 2018 9:53:13 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Runtime error caught during grid runnable execution: Socket reader [id=1711, name=tcp-disco-sock-reader-#11, nodeId=null] java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "tcp-disco-sock-reader-#11" java.lang.OutOfMemoryError: GC overhead limit exceeded Mar 07, 2018 9:53:24 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Runtime error caught during grid runnable execution: Socket reader [id=1710, name=tcp-disco-sock-reader-#10, nodeId=null] java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "tcp-disco-sock-reader-#10" java.lang.OutOfMemoryError: GC overhead limit exceeded Mar 07, 2018 9:54:45 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Runtime error caught during grid runnable execution: Socket reader [id=1712, name=tcp-disco-sock-reader-#12, nodeId=null] java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "tcp-disco-sock-reader-#12" java.lang.OutOfMemoryError: GC overhead limit exceeded Mar 07, 2018 9:54:52 AM org.apache.ignite.logger.java.JavaLogger error SEVERE: Runtime error caught during grid runnable execution: Socket reader [id=1713, name=tcp-disco-sock-reader-#13, nodeId=null] java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "tcp-disco-sock-reader-#13" java.lang.OutOfMemoryError: GC overhead limit exceeded {noformat} > Memory leak during creating/destroying local cache > -------------------------------------------------- > > Key: IGNITE-7319 > URL: https://issues.apache.org/jira/browse/IGNITE-7319 > Project: Ignite > Issue Type: Bug > Components: cache > Affects Versions: 2.3 > Reporter: Mikhail Cherkasov > Assignee: Alexey Goncharuk > Priority: Major > Fix For: 2.5 > > > The following code creates local caches: > {code} > private IgniteCache<String, BinaryObject> createLocalCache(String name) { > CacheConfiguration<String, BinaryObject> cCfg = new > CacheConfiguration<>(); > cCfg.setName(name); > cCfg.setGroupName("localCaches"); // without group leak is much > bigger! > cCfg.setStoreKeepBinary(true); > cCfg.setCacheMode(CacheMode.LOCAL); > cCfg.setOnheapCacheEnabled(false); > cCfg.setCopyOnRead(false); > cCfg.setBackups(0); > cCfg.setWriteBehindEnabled(false); > cCfg.setReadThrough(false); > cCfg.setReadFromBackup(false); > cCfg.setQueryEntities(<some basic query here>); > return ignite.createCache(cCfg).withKeepBinary(); > } > {code} > The caches are placed in the queue and are picked up by the worker thread > which just destroys them after removing from the queue. > This setup seems to generate a memory leak of about 1GB per day. > When looking at heap dump, I see all space is occupied by instances of > java.util.concurrent.ConcurrentSkipListMap$Node. > User list: > http://apache-ignite-users.70518.x6.nabble.com/Memory-leak-in-GridCachePartitionExchangeManager-tt18995.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)