Hi Shawn. It looks strange that each entry costs 10k in average, but you expect 50 bytes at most. You wrote that there are 300k entries, but I see 5000k keys in Heap.
Grid configuration and Heapdump would be helpful to understand who holds all that data. Or It would be great to have a reproducer. On Thu, Oct 27, 2016 at 10:29 AM, Shawn Du <[email protected]> wrote: > Hi Andrey Mashenkov, > > > > I checkout pr/1101 pr/1037 and have a test both of them. Things maybe go > better but not resolved. > > > > This is my cache state, I think it is full, for code like this: > > > > config.setSwapEnabled(swapEnable); > if (swapEnable) > { > EvictionPolicy policy = new FifoEvictionPolicy(10000); > config.setEvictionPolicy(policy); > } > > > > new caches should be in memory and the old will be evicted into disk. The > memory should stopping growing. > > After my test, it is not the case. It is still keep growing fast, at speed > of 500M/30minutes. > > I can understand that there will need some extra memory to maintain > something, but it leaks too fast. > > > > Thanks > > Shawn > > > > > > +-------------------------------------+-------------+------- > +-------------------------------+-----------+-----------+--- > --------+-----------+ > > | onlinesessioncount_by_cityid(@c1) | PARTITIONED | 2 | min: 0 (0 / > 0) | min: 0 | min: 0 | min: 0 | min: 0 | > > | | | | avg: > 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 | > > | | | | max: 10000 > (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 | > > +-------------------------------------+-------------+------- > +-------------------------------+-----------+-----------+--- > --------+-----------+ > > | onlinesessioncount_by_regionid(@c2) | PARTITIONED | 2 | min: 0 (0 / > 0) | min: 0 | min: 0 | min: 0 | min: 0 | > > | | | | avg: > 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 | > > | | | | max: 10000 > (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 | > > +-------------------------------------+-------------+------- > +-------------------------------+-----------+-----------+--- > --------+-----------+ > > | onlineviewcount_by_cityid(@c3) | PARTITIONED | 2 | min: 0 (0 / > 0) | min: 0 | min: 0 | min: 0 | min: 0 | > > | | | | avg: > 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 | > > | | | | max: 10000 > (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 | > > +-------------------------------------+-------------+------- > +-------------------------------+-----------+-----------+--- > --------+-----------+ > > | onlineviewcount_by_regionid(@c4) | PARTITIONED | 2 | min: 0 (0 / > 0) | min: 0 | min: 0 | min: 0 | min: 0 | > > | | | | avg: > 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 | > > | | | | max: 10000 > (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 | > > +-------------------------------------+-------------+------- > +-------------------------------+-----------+-----------+--- > --------+-----------+ > > | onlinevisitorcount_by_cityid(@c5) | PARTITIONED | 2 | min: 0 (0 / > 0) | min: 0 | min: 0 | min: 0 | min: 0 | > > | | | | avg: > 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 | > > | | | | max: 10000 > (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 | > > +-------------------------------------+-------------+------- > +-------------------------------+-----------+-----------+--- > --------+-----------+ > > | onlinevisitorcount_by_regionid(@c6) | PARTITIONED | 2 | min: 0 (0 / > 0) | min: 0 | min: 0 | min: 0 | min: 0 | > > | | | | avg: > 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 | > > | | | | max: 10000 > (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 | > > +-------------------------------------+-------------+------- > +-------------------------------+-----------+-----------+--- > --------+-----------+ > > | videoviewcount_by_cityid(@c7) | PARTITIONED | 2 | min: 0 (0 / > 0) | min: 0 | min: 0 | min: 0 | min: 0 | > > | | | | avg: 5000.00 > (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 | > > | | | | max: 10000 > (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 | > > +-------------------------------------+-------------+------- > +-------------------------------+-----------+-----------+--- > --------+-----------+ > > | videoviewcount_by_regionid(@c8) | PARTITIONED | 2 | min: 0 (0 / > 0) | min: 0 | min: 0 | min: 0 | min: 0 | > > | | | | avg: > 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 | > > | | | | max: 10000 > (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 | > > +----------------------------------------------------------- > ------------------------------------------------------------ > --------------------+ > > > > *发件人:* Shawn Du [mailto:[email protected]] > *发送时间:* 2016年10月27日 9:17 > *收件人:* [email protected] > *主题:* 答复: 答复: ignite used too much memory > > > > Hi Andrey Mashenkov, > > > > Thanks, I will have a try and apply these locally. > > > > Do you know 1.8’s release plan? > > > > Thanks > > Shawn > > > > *发件人:* Andrey Mashenkov [mailto:[email protected] > <[email protected]>] > *发送时间:* 2016年10月26日 20:53 > *收件人:* [email protected] > *主题:* Re: 答复: ignite used too much memory > > > > Hi, Shawn Du > > > > It seems you faced next 2 bugs. > > > > First bug: High memory utilization using OffHeap with ExpirePolicy. Issue > has a fix, but it is not merged to master yet. See: > https://issues.apache.org/jira/browse/IGNITE-3840. > > Second bug: TTL Manager continue track evicted (and removed) entries, > https://issues.apache.org/jira/browse/IGNITE-3948, it seems to be ok for > merge, but still is not present in master. > > > > You can try to merge them locally or wait until they will be available in > master. > > > > On Wed, Oct 26, 2016 at 2:35 PM, Shawn Du <[email protected]> wrote: > > Hi experts, > > > > Can anyone help to explain ignite memory model? > > > > Now I tried following ways, but no effect. > > 1) Remove all indexes. > > 2) enable swap. I see more than 800M’s data are stored in swapspace > directory. > > 3) Don’t cache short life entries. All cache with fifo evict policy > and max size is 10k. > > > > After running for a while, I have a check the caches, there were only 20k > entries, but the memory still grow up? > > > > Please help. > > > > Thanks > > Shawn > > > > > > *发件人**:* Shawn Du [mailto:[email protected]] > *发送时间**:* 2016年10月26日 17:17 > *收件人**:* [email protected] > *主题**:* 答复: ignite used too much memory > > > > Hi > > > > This is the output of jmap histo:live <pid>. > > > > Any useful information? There are about 300k cache entries. Also I enable > swap. > > > > Each entry costs 10K memory in average. But each entry only has 50 bytes > at most. > > > > Any help will be appreciated. > > > > num #instances #bytes class name > > ---------------------------------------------- > > 1: 21642240 519413760 org.apache.ignite.internal. > util.GridCircularBuffer$Item > > 2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node; > > 3: 6056069 304233144 [B > > 4: 5754322 279563064 [C > > 5: 11376526 273036624 java.util.concurrent. > ConcurrentSkipListMap$Node > > 6: 5688514 227540560 org.apache.ignite.internal. > processors.query.h2.opt.GridH2KeyValueRowOnheap > > 7: 5684293 181897376 org.apache.ignite.internal. > processors.cache.GridCacheTtlManager$EntryWrapper > > 8: 5333102 170659264 java.util.concurrent. > ConcurrentHashMap$Node > > 9: 5325117 170403744 org.apache.ignite.spi.swapspace.file. > FileSwapSpaceSpi$SwapValue > > 10: 5754728 138113472 java.lang.String > > 11: 5689577 136549848 java.util.concurrent. > ConcurrentSkipListMap$Index > > 12: 5686353 136472472 org.apache.ignite.internal. > processors.cache.KeyCacheObjectImpl > > 13: 5325117 127802808 org.apache.ignite.spi. > swapspace.SwapKey > > 14: 5690452 91047232 org.h2.value.ValueString > > 15: 85112 87930752 [Lorg.apache.ignite.internal. > util.GridCircularBuffer$Item; > > 16: 1515799 48505568 java.util.HashMap$Node > > 17: 268818 42628512 [Ljava.lang.Object; > > 18: 13467 35031504 [Ljava.util.concurrent. > ConcurrentHashMap$Node; > > 19: 361155 23113920 org.apache.ignite.internal. > processors.cache.distributed.dht.atomic.GridDhtAtomicCacheEntry > > 20: 1226922 19630752 java.lang.Integer > > 21: 102183 15207336 [Ljava.util.HashMap$Node; > > 22: 363397 14535880 org.apache.ignite.internal. > binary.BinaryObjectImpl > > 23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node > > 24: 359176 11493632 org.apache.ignite.internal. > processors.cache.extras.GridCacheTtlEntryExtras > > 25: 363397 8721528 org.apache.ignite.internal. > processors.query.h2.opt.GridH2ValueCacheObject > > 26: 257615 6182760 java.util.concurrent.atomic.AtomicLong > > 27: 170775 5464800 java.util.concurrent.locks. > ReentrantLock$NonfairSync > > 28: 85112 5447168 org.apache.ignite.internal. > processors.cache.distributed.dht.GridDhtLocalPartition > > 29: 85112 5447168 org.apache.ignite.internal. > processors.cache.distributed.dht.GridDhtLocalPartition$1 > > 30: 103188 4953024 java.util.HashMap > > 31: 87751 3510040 org.jsr166.ConcurrentHashMap8 > > 32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node > > 33: 90699 2902368 org.jsr166.LongAdder8 > > 34: 87277 2792864 java.lang.ref.WeakReference > > 35: 170663 2730608 java.util.concurrent.locks. > ReentrantLock > > 36: 85112 2723584 org.apache.ignite.internal. > processors.cache.GridCacheConcurrentMapImpl > > 37: 85112 2723584 org.apache.ignite.internal. > util.GridCircularBuffer > > 38: 134255 2148080 org.apache.ignite.internal. > processors.cache.CacheEvictableEntryImpl > > 39: 85304 2047296 java.util.concurrent. > CopyOnWriteArrayList > > 40: 10900 1693000 [Lorg.jsr166.ConcurrentLinkedHashMap$ > HashEntry; > > 41: 96594 1545504 java.util.HashSet > > 42: 86086 1377376 java.util.concurrent.atomic. > AtomicInteger > > 43: 85435 1366960 java.util.HashMap$KeySet > > 44: 85002 1360032 org.apache.ignite.internal. > processors.query.h2.opt.GridH2AbstractKeyValueRow$WeakValue > > 45: 44295 1063080 java.util.ArrayList > > 46: 13513 864832 java.util.concurrent.ConcurrentHashMap > > 47: 21090 843600 org.apache.ignite.internal. > processors.cache.version.GridCacheVersion > > 48: 160 653824 [Lorg.apache.ignite.internal. > processors.cache.distributed.dht.GridDhtLocalPartition; > > 49: 5664 637120 java.lang.Class > > 50: 7047 620136 java.lang.reflect.Method > > 51: 10900 610400 org.jsr166.ConcurrentLinkedHashMap$ > Segment > > 52: 12528 601344 java.util.concurrent.locks. > ReentrantReadWriteLock$NonfairSync > > 53: 11566 555168 org.h2.expression.ExpressionColumn > > 54: 16698 400752 java.lang.Long > > 55: 12522 400704 javax.management.MBeanAttributeInfo > > 56: 7878 381504 [I > > 57: 1767 310992 org.h2.command.dml.Select > > 58: 6162 295776 org.apache.ignite.internal. > processors.affinity.GridAffinityAssignment > > 59: 7223 288920 java.util.LinkedHashMap$Entry > > 60: 9276 222624 org.h2.expression.ValueExpression > > 61: 6912 221184 org.h2.expression.Comparison > > 62: 3042 219024 java.lang.reflect.Field > > 63: 12529 200464 java.util.concurrent.locks. > ReentrantReadWriteLock$ReadLock > > 64: 12529 200464 java.util.concurrent.locks. > ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter > > 65: 12529 200464 java.util.concurrent.locks. > ReentrantReadWriteLock$WriteLock > > 66: 1767 197904 org.h2.table.TableFilter > > 67: 2116 186208 org.apache.ignite.internal. > processors.jobmetrics.GridJobMetricsSnapshot > > 68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement > > 69: 5145 164640 org.h2.expression.ConditionAndOr > > 70: 7674 162472 [Ljava.lang.Class; > > 71: 4654 148928 org.h2.expression.Alias > > 72: 1726 137328 [S > > 73: 1767 127224 org.h2.index.IndexCursor > > 74: 7800 124800 java.lang.Object > > 75: 1360 119680 org.apache.ignite.internal.util. > StripedCompositeReadWriteLock$ReadLock > > 76: 1767 113088 org.h2.jdbc.JdbcResultSet > > 77: 1992 111552 java.util.LinkedHashMap > > 78: 4228 101472 org.h2.value.ValueLong > > 79: 2804 89728 java.lang.ThreadLocal$ > ThreadLocalMap$Entry > > 80: 309 88992 org.apache.ignite.configuration. > CacheConfiguration > > 81: 1767 84816 org.h2.command.CommandContainer > > 82: 1172 84384 org.h2.expression.Aggregate > > 83: 777 80808 org.h2.table.Column > > 84: 2502 80064 java.util.concurrent. > ConcurrentSkipListMap$HeadIndex > > 85: 2939 79912 [Lorg.h2.expression.Expression; > > 86: 1870 73152 [Lorg.h2.table.IndexColumn; > > 87: 1798 71920 java.lang.ref.SoftReference > > 88: 17 69904 [Ljava.nio.ByteBuffer; > > 89: 2166 69312 org.apache.ignite.internal. > GridLoggerProxy > > 90: 1715 68600 org.h2.expression.Operation > > 91: 64 66560 [Lorg.apache.ignite.internal. > processors.jobmetrics.GridJobMetricsSnapshot; > > 92: 20 65920 [Ljava.nio.channels.SelectionKey; > > > > > > > > > > *发件人**:* Shawn Du [mailto:[email protected] > <[email protected]>] > *发送时间**:* 2016年10月26日 13:25 > *收件人**:* [email protected] > *主题**:* ignite used too much memory > > > > Hi, > > > > In my ignite server, I have several caches, each cache has about 10k > entries. > > I build the entry using binary object. Each entry just has 3 or 4 fields, > each field is short, less than 20 bytes. But I enable index for each field. > > Most entry has set expired time. The expired time is short, about 90 > seconds. > > After run for 2 hours, 8G memory are used, and the ignite run out of > memory. > > I build the ignite from source code, and use yesterdays’ github master > branch code. > > > > My questions: > > 1) Does the expired data release the memory? > > 2) How ignite build the index? How much memory it cost? > > > > Thanks > > Shawn > > > > > > > > >
