Hi, 1.8 is supposed to be released in November. You may want to subscribe to the dev list and keep an eye on "Apache Ignite 1.8 Release” discussion going there.
— Denis > On Oct 26, 2016, at 6:16 PM, Shawn Du <[email protected]> wrote: > > Hi Andrey Mashenkov, > > Thanks, I will have a try and apply these locally. > > Do you know 1.8’s release plan? > > Thanks > Shawn > > 发件人: Andrey Mashenkov [mailto:[email protected]] > 发送时间: 2016年10月26日 20:53 > 收件人: [email protected] > 主题: Re: 答复: ignite used too much memory > > Hi, Shawn Du > > It seems you faced next 2 bugs. > > First bug: High memory utilization using OffHeap with ExpirePolicy. Issue has > a fix, but it is not merged to master yet. See: > https://issues.apache.org/jira/browse/IGNITE-3840 > <https://issues.apache.org/jira/browse/IGNITE-3840>. > Second bug: TTL Manager continue track evicted (and removed) entries, > https://issues.apache.org/jira/browse/IGNITE-3948 > <https://issues.apache.org/jira/browse/IGNITE-3948>, it seems to be ok for > merge, but still is not present in master. > > You can try to merge them locally or wait until they will be available in > master. > > On Wed, Oct 26, 2016 at 2:35 PM, Shawn Du <[email protected] > <mailto:[email protected]>> wrote: >> Hi experts, >> >> Can anyone help to explain ignite memory model? >> >> Now I tried following ways, but no effect. >> 1) Remove all indexes. >> >> 2) enable swap. I see more than 800M’s data are stored in swapspace >> directory. >> >> 3) Don’t cache short life entries. All cache with fifo evict policy >> and max size is 10k. >> >> >> After running for a while, I have a check the caches, there were only 20k >> entries, but the memory still grow up? >> >> Please help. >> >> Thanks >> Shawn >> >> >> 发件人: Shawn Du [mailto:[email protected] >> <mailto:[email protected]>] >> 发送时间: 2016年10月26日 17:17 >> 收件人: [email protected] <mailto:[email protected]> >> 主题: 答复: ignite used too much memory >> >> Hi >> >> This is the output of jmap histo:live <pid>. >> >> Any useful information? There are about 300k cache entries. Also I enable >> swap. >> >> Each entry costs 10K memory in average. But each entry only has 50 bytes at >> most. >> >> Any help will be appreciated. >> >> num #instances #bytes class name >> ---------------------------------------------- >> 1: 21642240 519413760 >> org.apache.ignite.internal.util.GridCircularBuffer$Item >> 2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node; >> 3: 6056069 304233144 [B >> 4: 5754322 279563064 [C >> 5: 11376526 273036624 >> java.util.concurrent.ConcurrentSkipListMap$Node >> 6: 5688514 227540560 >> org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap >> 7: 5684293 181897376 >> org.apache.ignite.internal.processors.cache.GridCacheTtlManager$EntryWrapper >> 8: 5333102 170659264 >> java.util.concurrent.ConcurrentHashMap$Node >> 9: 5325117 170403744 >> org.apache.ignite.spi.swapspace.file.FileSwapSpaceSpi$SwapValue >> 10: 5754728 138113472 java.lang.String >> 11: 5689577 136549848 >> java.util.concurrent.ConcurrentSkipListMap$Index >> 12: 5686353 136472472 >> org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl >> 13: 5325117 127802808 org.apache.ignite.spi.swapspace.SwapKey >> 14: 5690452 91047232 org.h2.value.ValueString >> 15: 85112 87930752 >> [Lorg.apache.ignite.internal.util.GridCircularBuffer$Item; >> 16: 1515799 48505568 java.util.HashMap$Node >> 17: 268818 42628512 [Ljava.lang.Object; >> 18: 13467 35031504 >> [Ljava.util.concurrent.ConcurrentHashMap$Node; >> 19: 361155 23113920 >> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCacheEntry >> 20: 1226922 19630752 java.lang.Integer >> 21: 102183 15207336 [Ljava.util.HashMap$Node; >> 22: 363397 14535880 >> org.apache.ignite.internal.binary.BinaryObjectImpl >> 23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node >> 24: 359176 11493632 >> org.apache.ignite.internal.processors.cache.extras.GridCacheTtlEntryExtras >> 25: 363397 8721528 >> org.apache.ignite.internal.processors.query.h2.opt.GridH2ValueCacheObject >> 26: 257615 6182760 java.util.concurrent.atomic.AtomicLong >> 27: 170775 5464800 >> java.util.concurrent.locks.ReentrantLock$NonfairSync >> 28: 85112 5447168 >> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition >> 29: 85112 5447168 >> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition$1 >> 30: 103188 4953024 java.util.HashMap >> 31: 87751 3510040 org.jsr166.ConcurrentHashMap8 >> 32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node >> 33: 90699 2902368 org.jsr166.LongAdder8 >> 34: 87277 2792864 java.lang.ref.WeakReference >> 35: 170663 2730608 java.util.concurrent.locks.ReentrantLock >> 36: 85112 2723584 >> org.apache.ignite.internal.processors.cache.GridCacheConcurrentMapImpl >> 37: 85112 2723584 >> org.apache.ignite.internal.util.GridCircularBuffer >> 38: 134255 2148080 >> org.apache.ignite.internal.processors.cache.CacheEvictableEntryImpl >> 39: 85304 2047296 java.util.concurrent.CopyOnWriteArrayList >> 40: 10900 1693000 >> [Lorg.jsr166.ConcurrentLinkedHashMap$HashEntry; >> 41: 96594 1545504 java.util.HashSet >> 42: 86086 1377376 java.util.concurrent.atomic.AtomicInteger >> 43: 85435 1366960 java.util.HashMap$KeySet >> 44: 85002 1360032 >> org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow$WeakValue >> 45: 44295 1063080 java.util.ArrayList >> 46: 13513 864832 java.util.concurrent.ConcurrentHashMap >> 47: 21090 843600 >> org.apache.ignite.internal.processors.cache.version.GridCacheVersion >> 48: 160 653824 >> [Lorg.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition; >> 49: 5664 637120 java.lang.Class >> 50: 7047 620136 java.lang.reflect.Method >> 51: 10900 610400 >> org.jsr166.ConcurrentLinkedHashMap$Segment >> 52: 12528 601344 >> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync >> 53: 11566 555168 org.h2.expression.ExpressionColumn >> 54: 16698 400752 java.lang.Long >> 55: 12522 400704 javax.management.MBeanAttributeInfo >> 56: 7878 381504 [I >> 57: 1767 310992 org.h2.command.dml.Select >> 58: 6162 295776 >> org.apache.ignite.internal.processors.affinity.GridAffinityAssignment >> 59: 7223 288920 java.util.LinkedHashMap$Entry >> 60: 9276 222624 org.h2.expression.ValueExpression >> 61: 6912 221184 org.h2.expression.Comparison >> 62: 3042 219024 java.lang.reflect.Field >> 63: 12529 200464 >> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock >> 64: 12529 200464 >> java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter >> 65: 12529 200464 >> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock >> 66: 1767 197904 org.h2.table.TableFilter >> 67: 2116 186208 >> org.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot >> 68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement >> 69: 5145 164640 org.h2.expression.ConditionAndOr >> 70: 7674 162472 [Ljava.lang.Class; >> 71: 4654 148928 org.h2.expression.Alias >> 72: 1726 137328 [S >> 73: 1767 127224 org.h2.index.IndexCursor >> 74: 7800 124800 java.lang.Object >> 75: 1360 119680 >> org.apache.ignite.internal.util.StripedCompositeReadWriteLock$ReadLock >> 76: 1767 113088 org.h2.jdbc.JdbcResultSet >> 77: 1992 111552 java.util.LinkedHashMap >> 78: 4228 101472 org.h2.value.ValueLong >> 79: 2804 89728 >> java.lang.ThreadLocal$ThreadLocalMap$Entry >> 80: 309 88992 >> org.apache.ignite.configuration.CacheConfiguration >> 81: 1767 84816 org.h2.command.CommandContainer >> 82: 1172 84384 org.h2.expression.Aggregate >> 83: 777 80808 org.h2.table.Column >> 84: 2502 80064 >> java.util.concurrent.ConcurrentSkipListMap$HeadIndex >> 85: 2939 79912 [Lorg.h2.expression.Expression; >> 86: 1870 73152 [Lorg.h2.table.IndexColumn; >> 87: 1798 71920 java.lang.ref.SoftReference >> 88: 17 69904 [Ljava.nio.ByteBuffer; >> 89: 2166 69312 >> org.apache.ignite.internal.GridLoggerProxy >> 90: 1715 68600 org.h2.expression.Operation >> 91: 64 66560 >> [Lorg.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot; >> 92: 20 65920 [Ljava.nio.channels.SelectionKey; >> >> >> >> >> 发件人: Shawn Du [mailto:[email protected] >> <mailto:[email protected]>] >> 发送时间: 2016年10月26日 13:25 >> 收件人: [email protected] <mailto:[email protected]> >> 主题: ignite used too much memory >> >> Hi, >> >> In my ignite server, I have several caches, each cache has about 10k >> entries. >> I build the entry using binary object. Each entry just has 3 or 4 fields, >> each field is short, less than 20 bytes. But I enable index for each field. >> Most entry has set expired time. The expired time is short, about 90 >> seconds. >> After run for 2 hours, 8G memory are used, and the ignite run out of memory. >> I build the ignite from source code, and use yesterdays’ github master >> branch code. >> >> My questions: >> 1) Does the expired data release the memory? >> >> 2) How ignite build the index? How much memory it cost? >> >> >> Thanks >> Shawn
