Hi Sam, Seems, Ignite can allocate memory for offheap cache map.
Have you tried to reduce CacheConfiguration.startSize property? It is 1_500_000 by default. On Fri, Jun 16, 2017 at 2:10 AM, [email protected] < [email protected]> wrote: > Hi, > > We are using Ignite 1.9, OFFHEAP with SWAP disabled. We are creating caches > programmatic and want to use SQL. > At one instance creating 350 empty cache ran into out-of-memory. We are > already setting low queue size for delete history > (IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE=100) > > Log metrics showing 25% free heap memory and 87% free off-heap. Can you > please help me understand this issue and possible resolution? OOM with heap > or offheap or H2 ? > > Below is Ignite log with stack trace as well as periodic metrics. > > ============= > 2017-06-15 11:41:41,404,[exchange-worker-#68%TESTNODE%],Skipping > rebalancing > (nothing scheduled) [top=AffinityTopologyVersion [topVer=1, > minorTopVer=348], evt=DISCOVERY_CUSTOM_EVT, > node=0c23715a-8c78-4852-9b1a-a5f101d65b7b] > 2017-06-15 11:41:43,345,[exchange-worker-#68%TESTNODE%],Failed to > reinitialize local partitions (preloading will be stopped): > GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1, > minorTopVer=349], nodeId=0c23715a, evt=DISCOVERY_CUSTOM_EVT] > class > org.apache.ignite.internal.util.offheap.GridOffHeapOutOfMemoryException: > Failed to allocate memory [total=0, failed=16384] > at > org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocate0( > GridUnsafeMemory.java:182) > at > org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory. > allocateSystem(GridUnsafeMemory.java:145) > at > org.apache.ignite.internal.util.offheap.unsafe. > GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:624) > at > org.apache.ignite.internal.util.offheap.unsafe. > GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:590) > at > org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.init( > GridUnsafeMap.java:273) > at > org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.<init>( > GridUnsafeMap.java:248) > at > org.apache.ignite.internal.util.offheap.unsafe.GridUnsafePartitionedMap.< > init>(GridUnsafePartitionedMap.java:111) > at > org.apache.ignite.internal.util.offheap.GridOffHeapMapFactory. > unsafePartitionedMap(GridOffHeapMapFactory.java:224) > at > org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor.create( > GridOffHeapProcessor.java:72) > at > org.apache.ignite.internal.processors.cache.GridCacheSwapManager. > initOffHeap(GridCacheSwapManager.java:251) > at > org.apache.ignite.internal.processors.cache.GridCacheSwapManager.start0( > GridCacheSwapManager.java:138) > at > org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.start( > GridCacheManagerAdapter.java:50) > at > org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache( > GridCacheProcessor.java:1091) > at > org.apache.ignite.internal.processors.cache.GridCacheProcessor. > prepareCacheStart(GridCacheProcessor.java:1745) > at > org.apache.ignite.internal.processors.cache.GridCacheProcessor. > prepareCacheStart(GridCacheProcessor.java:1636) > at > org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager. > onCacheChangeRequest(CacheAffinitySharedManager.java:382) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader. > GridDhtPartitionsExchangeFuture.onCacheChangeRequest( > GridDhtPartitionsExchangeFuture.java:581) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader. > GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFutur > e.java:464) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeMana > ger$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1674) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:745) > 2017-06-15 11:41:48,111,[grid-timeout-worker-#54%TESTNODE%], > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=0c23715a, name=TESTNODE, uptime=00:10:06:422] > ^-- H/N/C [hosts=1, nodes=1, CPUs=2] > ^-- CPU [cur=0.27%, avg=20.87%, GC=0%] > ^-- Heap [used=1461MB, free=26.63%, comm=1991MB] > ^-- Non heap [used=195MB, free=87.16%, comm=208MB] > ^-- Public thread pool [active=0, idle=0, qSize=0] > ^-- System thread pool [active=0, idle=6, qSize=0] > ^-- Outbound messages queue [size=0] > 2017-06-15 11:41:48,333,[exchange-worker-#68%TESTNODE%],Failed to wait for > completion of partition map exchange (preloading will not start): > GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, > reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null, > affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=349], > super=DiscoveryEvent [evtNode=TcpDiscoveryNode > [id=0c23715a-8c78-4852-9b1a-a5f101d65b7b, addrs=[127.0.0.1], > sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1, intOrder=1, > lastExchangeTime=1497506480276, loc=true, ver=1.9.0#20170302-sha1: > a8169d0a, > isClient=false], topVer=1, nodeId8=0c23715a, msg=null, > type=DISCOVERY_CUSTOM_EVT, tstamp=1497507101444]], crd=TcpDiscoveryNode > [id=0c23715a-8c78-4852-9b1a-a5f101d65b7b, addrs=[127.0.0.1], > sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1, intOrder=1, > lastExchangeTime=1497506480276, loc=true, ver=1.9.0#20170302-sha1: > a8169d0a, > isClient=false], exchId=GridDhtPartitionExchangeId > [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=349], > nodeId=0c23715a, evt=DISCOVERY_CUSTOM_EVT], added=true, > initFut=GridFutureAdapter [resFlag=2, res=false, startTime=1497507101444, > endTime=1497507107577, ignoreInterrupts=false, state=DONE], init=false, > topSnapshot=null, lastVer=null, partReleaseFut=null, affChangeMsg=null, > skipPreload=false, clientOnlyExchange=false, initTs=1497507101444, > centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], > srvNodes=[TcpDiscoveryNode [id=0c23715a-8c78-4852-9b1a-a5f101d65b7b, > addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:49500], discPort=49500, order=1, > intOrder=1, lastExchangeTime=1497506480276, loc=true, > ver=1.9.0#20170302-sha1:a8169d0a, isClient=false]], > super=GridFutureAdapter > [resFlag=1, res=class o.a.i.i.util.offheap.GridOffHeapOutOfMemoryExceptio > n: > Failed to allocate memory [total=0, failed=16384], startTime=1497507101444, > endTime=1497507107577, ignoreInterrupts=false, state=DONE]] > class org.apache.ignite.IgniteCheckedException: Failed to allocate memory > [total=0, failed=16384] > at org.apache.ignite.internal.util.IgniteUtils.cast( > IgniteUtils.java:7239) > at > org.apache.ignite.internal.util.future.GridFutureAdapter. > get0(GridFutureAdapter.java:198) > at > org.apache.ignite.internal.util.future.GridFutureAdapter. > get(GridFutureAdapter.java:139) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeMana > ger$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1680) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:745) > Caused by: class > org.apache.ignite.internal.util.offheap.GridOffHeapOutOfMemoryException: > Failed to allocate memory [total=0, failed=16384] > at > org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory.allocate0( > GridUnsafeMemory.java:182) > at > org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory. > allocateSystem(GridUnsafeMemory.java:145) > at > org.apache.ignite.internal.util.offheap.unsafe. > GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:624) > at > org.apache.ignite.internal.util.offheap.unsafe. > GridUnsafeMap$Segment.<init>(GridUnsafeMap.java:590) > at > org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.init( > GridUnsafeMap.java:273) > at > org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMap.<init>( > GridUnsafeMap.java:248) > at > org.apache.ignite.internal.util.offheap.unsafe.GridUnsafePartitionedMap.< > init>(GridUnsafePartitionedMap.java:111) > at > org.apache.ignite.internal.util.offheap.GridOffHeapMapFactory. > unsafePartitionedMap(GridOffHeapMapFactory.java:224) > at > org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor.create( > GridOffHeapProcessor.java:72) > at > org.apache.ignite.internal.processors.cache.GridCacheSwapManager. > initOffHeap(GridCacheSwapManager.java:251) > at > org.apache.ignite.internal.processors.cache.GridCacheSwapManager.start0( > GridCacheSwapManager.java:138) > at > org.apache.ignite.internal.processors.cache.GridCacheManagerAdapter.start( > GridCacheManagerAdapter.java:50) > at > org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCache( > GridCacheProcessor.java:1091) > at > org.apache.ignite.internal.processors.cache.GridCacheProcessor. > prepareCacheStart(GridCacheProcessor.java:1745) > at > org.apache.ignite.internal.processors.cache.GridCacheProcessor. > prepareCacheStart(GridCacheProcessor.java:1636) > at > org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager. > onCacheChangeRequest(CacheAffinitySharedManager.java:382) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader. > GridDhtPartitionsExchangeFuture.onCacheChangeRequest( > GridDhtPartitionsExchangeFuture.java:581) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader. > GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFutur > e.java:464) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeMana > ger$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1674) > ... 2 more > > ============= > > Thanks, > -Sam > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Out-Of-Memory-tp13829.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Best regards, Andrey V. Mashenkov
