Hi Andrey Mashenkov,
I checkout pr/1101 pr/1037 and have a test both of them. Things maybe go better
but not resolved.
This is my cache state, I think it is full, for code like this:
config.setSwapEnabled(swapEnable);
if (swapEnable)
{
EvictionPolicy policy = new FifoEvictionPolicy(10000);
config.setEvictionPolicy(policy);
}
new caches should be in memory and the old will be evicted into disk. The
memory should stopping growing.
After my test, it is not the case. It is still keep growing fast, at speed of
500M/30minutes.
I can understand that there will need some extra memory to maintain something,
but it leaks too fast.
Thanks
Shawn
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlinesessioncount_by_cityid(@c1) | PARTITIONED | 2 | min: 0 (0 / 0)
| min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00
(5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000
/ 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlinesessioncount_by_regionid(@c2) | PARTITIONED | 2 | min: 0 (0 / 0)
| min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00
(5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000
/ 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlineviewcount_by_cityid(@c3) | PARTITIONED | 2 | min: 0 (0 / 0)
| min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00
(5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000
/ 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlineviewcount_by_regionid(@c4) | PARTITIONED | 2 | min: 0 (0 / 0)
| min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00
(5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000
/ 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlinevisitorcount_by_cityid(@c5) | PARTITIONED | 2 | min: 0 (0 / 0)
| min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00
(5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000
/ 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlinevisitorcount_by_regionid(@c6) | PARTITIONED | 2 | min: 0 (0 / 0)
| min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00
(5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000
/ 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| videoviewcount_by_cityid(@c7) | PARTITIONED | 2 | min: 0 (0 / 0)
| min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00
(5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000
/ 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| videoviewcount_by_regionid(@c8) | PARTITIONED | 2 | min: 0 (0 / 0)
| min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00
(5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000
/ 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------------------------------------------------------------------------------------------------------------+
发件人: Shawn Du [mailto:[email protected]]
发送时间: 2016年10月27日 9:17
收件人: [email protected]
主题: 答复: 答复: ignite used too much memory
Hi Andrey Mashenkov,
Thanks, I will have a try and apply these locally.
Do you know 1.8’s release plan?
Thanks
Shawn
发件人: Andrey Mashenkov [ <mailto:[email protected]>
mailto:[email protected]]
发送时间: 2016年10月26日 20:53
收件人: <mailto:[email protected]> [email protected]
主题: Re: 答复: ignite used too much memory
Hi, Shawn Du
It seems you faced next 2 bugs.
First bug: High memory utilization using OffHeap with ExpirePolicy. Issue has a
fix, but it is not merged to master yet. See:
<https://issues.apache.org/jira/browse/IGNITE-3840>
https://issues.apache.org/jira/browse/IGNITE-3840.
Second bug: TTL Manager continue track evicted (and removed) entries,
<https://issues.apache.org/jira/browse/IGNITE-3948>
https://issues.apache.org/jira/browse/IGNITE-3948, it seems to be ok for merge,
but still is not present in master.
You can try to merge them locally or wait until they will be available in
master.
On Wed, Oct 26, 2016 at 2:35 PM, Shawn Du <[email protected]
<mailto:[email protected]> > wrote:
Hi experts,
Can anyone help to explain ignite memory model?
Now I tried following ways, but no effect.
1) Remove all indexes.
2) enable swap. I see more than 800M’s data are stored in swapspace
directory.
3) Don’t cache short life entries. All cache with fifo evict policy and
max size is 10k.
After running for a while, I have a check the caches, there were only 20k
entries, but the memory still grow up?
Please help.
Thanks
Shawn
发件人: Shawn Du [mailto: <mailto:[email protected]>
[email protected]]
发送时间: 2016年10月26日 17:17
收件人: <mailto:[email protected]> [email protected]
主题: 答复: ignite used too much memory
Hi
This is the output of jmap histo:live <pid>.
Any useful information? There are about 300k cache entries. Also I enable
swap.
Each entry costs 10K memory in average. But each entry only has 50 bytes at
most.
Any help will be appreciated.
num #instances #bytes class name
----------------------------------------------
1: 21642240 519413760
org.apache.ignite.internal.util.GridCircularBuffer$Item
2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node;
3: 6056069 304233144 [B
4: 5754322 279563064 [C
5: 11376526 273036624
java.util.concurrent.ConcurrentSkipListMap$Node
6: 5688514 227540560
org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap
7: 5684293 181897376
org.apache.ignite.internal.processors.cache.GridCacheTtlManager$EntryWrapper
8: 5333102 170659264 java.util.concurrent.ConcurrentHashMap$Node
9: 5325117 170403744
org.apache.ignite.spi.swapspace.file.FileSwapSpaceSpi$SwapValue
10: 5754728 138113472 java.lang.String
11: 5689577 136549848
java.util.concurrent.ConcurrentSkipListMap$Index
12: 5686353 136472472
org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl
13: 5325117 127802808 org.apache.ignite.spi.swapspace.SwapKey
14: 5690452 91047232 org.h2.value.ValueString
15: 85112 87930752
[Lorg.apache.ignite.internal.util.GridCircularBuffer$Item;
16: 1515799 48505568 java.util.HashMap$Node
17: 268818 42628512 [Ljava.lang.Object;
18: 13467 35031504
[Ljava.util.concurrent.ConcurrentHashMap$Node;
19: 361155 23113920
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCacheEntry
20: 1226922 19630752 java.lang.Integer
21: 102183 15207336 [Ljava.util.HashMap$Node;
22: 363397 14535880
org.apache.ignite.internal.binary.BinaryObjectImpl
23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node
24: 359176 11493632
org.apache.ignite.internal.processors.cache.extras.GridCacheTtlEntryExtras
25: 363397 8721528
org.apache.ignite.internal.processors.query.h2.opt.GridH2ValueCacheObject
26: 257615 6182760 java.util.concurrent.atomic.AtomicLong
27: 170775 5464800
java.util.concurrent.locks.ReentrantLock$NonfairSync
28: 85112 5447168
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition
29: 85112 5447168
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition$1
30: 103188 4953024 java.util.HashMap
31: 87751 3510040 org.jsr166.ConcurrentHashMap8
32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node
33: 90699 2902368 org.jsr166.LongAdder8
34: 87277 2792864 java.lang.ref.WeakReference
35: 170663 2730608 java.util.concurrent.locks.ReentrantLock
36: 85112 2723584
org.apache.ignite.internal.processors.cache.GridCacheConcurrentMapImpl
37: 85112 2723584
org.apache.ignite.internal.util.GridCircularBuffer
38: 134255 2148080
org.apache.ignite.internal.processors.cache.CacheEvictableEntryImpl
39: 85304 2047296 java.util.concurrent.CopyOnWriteArrayList
40: 10900 1693000
[Lorg.jsr166.ConcurrentLinkedHashMap$HashEntry;
41: 96594 1545504 java.util.HashSet
42: 86086 1377376 java.util.concurrent.atomic.AtomicInteger
43: 85435 1366960 java.util.HashMap$KeySet
44: 85002 1360032
org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow$WeakValue
45: 44295 1063080 java.util.ArrayList
46: 13513 864832 java.util.concurrent.ConcurrentHashMap
47: 21090 843600
org.apache.ignite.internal.processors.cache.version.GridCacheVersion
48: 160 653824
[Lorg.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition;
49: 5664 637120 java.lang.Class
50: 7047 620136 java.lang.reflect.Method
51: 10900 610400 org.jsr166.ConcurrentLinkedHashMap$Segment
52: 12528 601344
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
53: 11566 555168 org.h2.expression.ExpressionColumn
54: 16698 400752 java.lang.Long
55: 12522 400704 javax.management.MBeanAttributeInfo
56: 7878 381504 [I
57: 1767 310992 org.h2.command.dml.Select
58: 6162 295776
org.apache.ignite.internal.processors.affinity.GridAffinityAssignment
59: 7223 288920 java.util.LinkedHashMap$Entry
60: 9276 222624 org.h2.expression.ValueExpression
61: 6912 221184 org.h2.expression.Comparison
62: 3042 219024 java.lang.reflect.Field
63: 12529 200464
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
64: 12529 200464
java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
65: 12529 200464
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
66: 1767 197904 org.h2.table.TableFilter
67: 2116 186208
org.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot
68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement
69: 5145 164640 org.h2.expression.ConditionAndOr
70: 7674 162472 [Ljava.lang.Class;
71: 4654 148928 org.h2.expression.Alias
72: 1726 137328 [S
73: 1767 127224 org.h2.index.IndexCursor
74: 7800 124800 java.lang.Object
75: 1360 119680
org.apache.ignite.internal.util.StripedCompositeReadWriteLock$ReadLock
76: 1767 113088 org.h2.jdbc.JdbcResultSet
77: 1992 111552 java.util.LinkedHashMap
78: 4228 101472 org.h2.value.ValueLong
79: 2804 89728 java.lang.ThreadLocal$ThreadLocalMap$Entry
80: 309 88992
org.apache.ignite.configuration.CacheConfiguration
81: 1767 84816 org.h2.command.CommandContainer
82: 1172 84384 org.h2.expression.Aggregate
83: 777 80808 org.h2.table.Column
84: 2502 80064
java.util.concurrent.ConcurrentSkipListMap$HeadIndex
85: 2939 79912 [Lorg.h2.expression.Expression;
86: 1870 73152 [Lorg.h2.table.IndexColumn;
87: 1798 71920 java.lang.ref.SoftReference
88: 17 69904 [Ljava.nio.ByteBuffer;
89: 2166 69312 org.apache.ignite.internal.GridLoggerProxy
90: 1715 68600 org.h2.expression.Operation
91: 64 66560
[Lorg.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot;
92: 20 65920 [Ljava.nio.channels.SelectionKey;
发件人: Shawn Du [ <mailto:[email protected]>
mailto:[email protected]]
发送时间: 2016年10月26日 13:25
收件人: <mailto:[email protected]> [email protected]
主题: ignite used too much memory
Hi,
In my ignite server, I have several caches, each cache has about 10k entries.
I build the entry using binary object. Each entry just has 3 or 4 fields, each
field is short, less than 20 bytes. But I enable index for each field.
Most entry has set expired time. The expired time is short, about 90 seconds.
After run for 2 hours, 8G memory are used, and the ignite run out of memory.
I build the ignite from source code, and use yesterdays’ github master branch
code.
My questions:
1) Does the expired data release the memory?
2) How ignite build the index? How much memory it cost?
Thanks
Shawn
