Hi Shawn, > but in my ignite config ss.xml, I use another bean id "ignite.cfg" for class > org.apache.ignite.configuration.IgniteConfiguration. > will this cause the problem?
Yes, the above should be a reason on why the memory configuration changes were not picked up. — Denis > On Dec 13, 2017, at 7:05 PM, shawn.du <[email protected]> wrote: > > > Hi Denis, > [policyName=default, size=12.9 GB] > It seems that 30G off heap setting didn't take effect. > > I set it in default-config.xml > > <bean id="grid.cfg" > class="org.apache.ignite.configuration.IgniteConfiguration"> > <!-- Redefining maximum memory size for the cluster node usage. --> > <property name="dataStorageConfiguration"> > <bean > class="org.apache.ignite.configuration.DataStorageConfiguration"> > <!-- Redefining the default region's settings --> > <property name="defaultDataRegionConfiguration"> > <bean > class="org.apache.ignite.configuration.DataRegionConfiguration"> > <property name="name" value="Default_Region"/> > <!-- Setting the size of the default region to 30GB. > --> > <property name="maxSize" value="#{30L * 1024 * 1024 * > 1024}"/> > </bean> > </property> > </bean> > </property> > <!-- The rest of the parameters. --> > </bean> > > but in my ignite config ss.xml, I use another bean id "ignite.cfg" for class > org.apache.ignite.configuration.IgniteConfiguration. > will this cause the problem? > > Thanks > Shawn > > On 12/14/2017 10:48，shawn.du<[email protected]> > <mailto:[email protected]> wrote： > Hi Denis, > > after run 20 hours, ignite client can't update cache and threw blow > exceptions: > > Caused by: org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Not > enough memory allocated (consider increasing data region size or enabling > evictions) [policyName=default, size=12.9 GB] > at > org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:292) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.DataStructure.allocatePageNoReuse(DataStructure.java:117) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.DataStructure.allocatePage(DataStructure.java:105) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8400(BPlusTree.java:81) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.insertWithSplit(BPlusTree.java:2703) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.insert(BPlusTree.java:2665) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.access$2500(BPlusTree.java:2547) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:411) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:392) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:342) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:261) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$11100(BPlusTree.java:81) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.tryInsert(BPlusTree.java:2859) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.access$7600(BPlusTree.java:2547) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2285) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266) > ~[stormjar.jar:?] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006) > ~[stormjar.jar:?] > > I had only a node running ignite server. others are all in client mode. > I start ignite server with -Xms10g -Xmx10g and set default region size 30G. > disable persistence, disable on-heap cache. > > also find this using visorcmd with command "node " > | Maximum active jobs | 8 > | > | Current active jobs | 1 > | > | Average active jobs | 0.32 > | > | Maximum waiting jobs | 0 > | > | Current waiting jobs | 0 > | > | Average waiting jobs | 0.00 > | > | Maximum rejected jobs | 0 > | > | Current rejected jobs | 0 > | > | Average rejected jobs | 0.00 > | > | Maximum cancelled jobs | 0 > | > | Current cancelled jobs | 0 > | > | Average cancelled jobs | 0.00 > | > | Total rejected jobs | 0 > | > | Total executed jobs | 7038798 > | > | Total cancelled jobs | 0 > | > | Maximum job wait time | 0ms > | > | Current job wait time | 0ms > | > | Average job wait time | 0.00ms > | > | Maximum job execute time | 5232ms > | > | Curent job execute time | 0ms > | > | Average job execute time | 0.03ms > | > | Total busy time | 11731204ms > | > | Busy time % | 14.09% > | > | Current CPU load % | 0.80% > | > | Average CPU load % | 1.34% > | > | Heap memory initialized | 10gb > | > | Heap memory used | 7gb > | > | Heap memory committed | 10gb > | > | Heap memory maximum | 10gb > | > | Non-heap memory initialized | 2mb > | > | Non-heap memory used | 102mb > | > | Non-heap memory committed | 106mb > | > | Non-heap memory maximum | 0 > | > | Current thread count | 99 > | > | Maximum thread count | 104 > | > | Total started thread count | 12725 > | > | Current daemon thread count | 12 > > "Heap memory committed " is 10g, does it mean it is already full? I don't > know who eat up the 10G on heap memory. > I confirmed that all cached are stored in off-heap by visorcmd "cache". > anything I miss? > > Thanks > Shawn > > On 12/14/2017 07:56，Denis Magda<[email protected]> <mailto:[email protected]> > wrote： > Shawn, good to hear this, > > Just in case here is you can find some tips and tricks on memory and garbage > collection tuning for Ignite: > https://apacheignite.readme.io/docs/durable-memory-tuning > <https://apacheignite.readme.io/docs/durable-memory-tuning>https://apacheignite.readme.io/docs/jvm-and-system-tuning > <https://apacheignite.readme.io/docs/jvm-and-system-tuning> > > — > Denis > >> On Dec 12, 2017, at 10:50 PM, shawn.du <[email protected] >> <mailto:[email protected]>> wrote: >> >> Thank you Denis, >> >> I think OOM is caused by invalid configuration. for I supposed ignite use >> on-heap memory, I set a very big value for >> -Xmx -Xms, in this way, memory left for off-heap and other programs is >> limited. >> We are run testing now, till now all seems good. >> >> Thanks >> Shawn >> >> On 12/13/2017 07:36，Denis Magda<[email protected]> >> <mailto:[email protected]> wrote： >> Shawn, >> >> If you still need to use the on-heap caching by some reason make sure you >> enabled an eviction policy: >> https://apacheignite.readme.io/docs/evictions#section-java-heap-cache >> <https://apacheignite.readme.io/docs/evictions#section-java-heap-cache> >> >> Otherwise the on-heap cache will grow endlessly. >> >> BTW, what kind of OOM you got? It might be off-heap or on-heap caching >> related. Share the whole stack trace. >> >> — >> Denis >> >>> On Dec 12, 2017, at 4:37 AM, slava.koptilin <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hi Shawn, >>> >>>> how to disable off heap completely? >>> You cannot disable off-heap. As of 2.0, Apache Ignite stores all the data >>> outside of java heap. >>> >>>> does it mean half in on-heap and half in off-heap? >>> On-heap caching allows to get a subset of the data into java heap >>> and can be useful for scenarios when you do a lot of cache reads on server >>> nodes that work with cache entries in the binary form >>> or invoke cache entry deserialization. >>> >>> Thanks! >>> >>> >>> >>> -- >>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >>> <http://apache-ignite-users.70518.x6.nabble.com/> >> >> > >
