Hi Denis,
[policyName=default, size=12.9 GB]
It seems that 30G off heap setting didn't take effect.
I set it in default-config.xml
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Redefining maximum memory size for the cluster node usage. -->
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<!-- Redefining the default region's settings -->
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Default_Region"/>
<!-- Setting the size of the default region to 30GB. -->
<property name="maxSize" value="#{30L * 1024 * 1024 * 1024}"/>
</bean>
</property>
</bean>
</property>
<!-- The rest of the parameters. -->
</bean>
but in my ignite config ss.xml, I use another bean id "ignite.cfg" for class org.apache.ignite.configuration.IgniteConfiguration.
will this cause the problem?
Thanks
Shawn
On 12/14/2017 10:48，shawn.du<[email protected]> wrote：
Hi Denis,after run 20 hours, ignite client can't update cache and threw blow exceptions:Caused by: org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Not enough memory allocated (consider increasing data region size or enabling evictions) [policyName=default, size=12.9 GB]at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:292) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.DataStructure.allocatePageNoReuse(DataStructure.java:117) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.DataStructure.allocatePage(DataStructure.java:105) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8400(BPlusTree.java:81) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.insertWithSplit(BPlusTree.java:2703) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.insert(BPlusTree.java:2665) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.access$2500(BPlusTree.java:2547) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:411) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:392) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:342) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:261) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$11100(BPlusTree.java:81) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.tryInsert(BPlusTree.java:2859) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.access$7600(BPlusTree.java:2547) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2285) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266) ~[stormjar.jar:?]at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006) ~[stormjar.jar:?]I had only a node running ignite server. others are all in client mode.I start ignite server with -Xms10g -Xmx10g and set default region size 30G. disable persistence, disable on-heap cache.also find this using visorcmd with command "node "| Maximum active jobs | 8 || Current active jobs | 1 || Average active jobs | 0.32 || Maximum waiting jobs | 0 || Current waiting jobs | 0 || Average waiting jobs | 0.00 || Maximum rejected jobs | 0 || Current rejected jobs | 0 || Average rejected jobs | 0.00 || Maximum cancelled jobs | 0 || Current cancelled jobs | 0 || Average cancelled jobs | 0.00 || Total rejected jobs | 0 || Total executed jobs | 7038798 || Total cancelled jobs | 0 || Maximum job wait time | 0ms || Current job wait time | 0ms || Average job wait time | 0.00ms || Maximum job execute time | 5232ms || Curent job execute time | 0ms || Average job execute time | 0.03ms || Total busy time | 11731204ms || Busy time % | 14.09% || Current CPU load % | 0.80% || Average CPU load % | 1.34% || Heap memory initialized | 10gb || Heap memory used | 7gb || Heap memory committed | 10gb || Heap memory maximum | 10gb || Non-heap memory initialized | 2mb || Non-heap memory used | 102mb || Non-heap memory committed | 106mb || Non-heap memory maximum | 0 || Current thread count | 99 || Maximum thread count | 104 || Total started thread count | 12725 || Current daemon thread count | 12"Heap memory committed " is 10g, does it mean it is already full? I don't know who eat up the 10G on heap memory.I confirmed that all cached are stored in off-heap by visorcmd "cache". anything I miss?ThanksShawn
On 12/14/2017 07:56，Denis Magda<[email protected]> wrote：Shawn, good to hear this,Just in case here is you can find some tips and tricks on memory and garbage collection tuning for Ignite:https://apacheignite.readme.io/docs/jvm-and-system-tuning—DenisOn Dec 12, 2017, at 10:50 PM, shawn.du <[email protected]> wrote:Thank you Denis,I think OOM is caused by invalid configuration. for I supposed ignite use on-heap memory, I set a very big value for-Xmx -Xms, in this way, memory left for off-heap and other programs is limited.We are run testing now, till now all seems good.ThanksShawnShawn,If you still need to use the on-heap caching by some reason make sure you enabled an eviction policy:Otherwise the on-heap cache will grow endlessly.BTW, what kind of OOM you got? It might be off-heap or on-heap caching related. Share the whole stack trace.—DenisOn Dec 12, 2017, at 4:37 AM, slava.koptilin <[email protected]> wrote:Hi Shawn,how to disable off heap completely?You cannot disable off-heap. As of 2.0, Apache Ignite stores all the data
outside of java heap.does it mean half in on-heap and half in off-heap?On-heap caching allows to get a subset of the data into java heap
and can be useful for scenarios when you do a lot of cache reads on server
nodes that work with cache entries in the binary form
or invoke cache entry deserialization.
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
