Hi Denis,
after run 20 hours, ignite client can't update cache and threw blow exceptions:
Caused by: org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Not enough memory allocated (consider increasing data region size or enabling evictions) [policyName=default, size=12.9 GB]
at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:292) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.DataStructure.allocatePageNoReuse(DataStructure.java:117) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.DataStructure.allocatePage(DataStructure.java:105) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8400(BPlusTree.java:81) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.insertWithSplit(BPlusTree.java:2703) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.insert(BPlusTree.java:2665) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.access$2500(BPlusTree.java:2547) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:411) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:392) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:342) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:261) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$11100(BPlusTree.java:81) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.tryInsert(BPlusTree.java:2859) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.access$7600(BPlusTree.java:2547) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2285) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2266) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2006) ~[stormjar.jar:?]
I had only a node running ignite server. others are all in client mode.
I start ignite server with -Xms10g -Xmx10g and set default region size 30G. disable persistence, disable on-heap cache.
also find this using visorcmd with command "node "
| Maximum active jobs | 8 |
| Current active jobs | 1 |
| Average active jobs | 0.32 |
| Maximum waiting jobs | 0 |
| Current waiting jobs | 0 |
| Average waiting jobs | 0.00 |
| Maximum rejected jobs | 0 |
| Current rejected jobs | 0 |
| Average rejected jobs | 0.00 |
| Maximum cancelled jobs | 0 |
| Current cancelled jobs | 0 |
| Average cancelled jobs | 0.00 |
| Total rejected jobs | 0 |
| Total executed jobs | 7038798 |
| Total cancelled jobs | 0 |
| Maximum job wait time | 0ms |
| Current job wait time | 0ms |
| Average job wait time | 0.00ms |
| Maximum job execute time | 5232ms |
| Curent job execute time | 0ms |
| Average job execute time | 0.03ms |
| Total busy time | 11731204ms |
| Busy time % | 14.09% |
| Current CPU load % | 0.80% |
| Average CPU load % | 1.34% |
| Heap memory initialized | 10gb |
| Heap memory used | 7gb |
| Heap memory committed | 10gb |
| Heap memory maximum | 10gb |
| Non-heap memory initialized | 2mb |
| Non-heap memory used | 102mb |
| Non-heap memory committed | 106mb |
| Non-heap memory maximum | 0 |
| Current thread count | 99 |
| Maximum thread count | 104 |
| Total started thread count | 12725 |
| Current daemon thread count | 12
"Heap memory committed " is 10g, does it mean it is already full? I don't know who eat up the 10G on heap memory.
I confirmed that all cached are stored in off-heap by visorcmd "cache". anything I miss?
Thanks
Shawn
On 12/14/2017 07:56，Denis Magda<[email protected]> wrote：
Shawn, good to hear this,Just in case here is you can find some tips and tricks on memory and garbage collection tuning for Ignite:https://apacheignite.readme.io/docs/jvm-and-system-tuning—DenisOn Dec 12, 2017, at 10:50 PM, shawn.du <[email protected]> wrote:Thank you Denis,I think OOM is caused by invalid configuration. for I supposed ignite use on-heap memory, I set a very big value for-Xmx -Xms, in this way, memory left for off-heap and other programs is limited.We are run testing now, till now all seems good.ThanksShawnShawn,If you still need to use the on-heap caching by some reason make sure you enabled an eviction policy:Otherwise the on-heap cache will grow endlessly.BTW, what kind of OOM you got? It might be off-heap or on-heap caching related. Share the whole stack trace.—DenisOn Dec 12, 2017, at 4:37 AM, slava.koptilin <[email protected]> wrote:Hi Shawn,how to disable off heap completely?You cannot disable off-heap. As of 2.0, Apache Ignite stores all the data
outside of java heap.does it mean half in on-heap and half in off-heap?On-heap caching allows to get a subset of the data into java heap
and can be useful for scenarios when you do a lot of cache reads on server
nodes that work with cache entries in the binary form
or invoke cache entry deserialization.
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
