[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017184#comment-16017184 ] ASF GitHub Bot commented on IGNITE-4534: Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/1758 > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Alexey Goncharuk > Labels: important > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961820#comment-15961820 ] ASF GitHub Bot commented on IGNITE-4534: Github user glukos closed the pull request at: https://github.com/apache/ignite/pull/1731 > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Labels: important > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961819#comment-15961819 ] ASF GitHub Bot commented on IGNITE-4534: Github user glukos closed the pull request at: https://github.com/apache/ignite/pull/1674 > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Labels: important > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956532#comment-15956532 ] Semen Boikov commented on IGNITE-4534: -- Ivan, I'm reviewing your changes, meanwhile I want to know if you tested evictions impact on performance: - did you compare performance before/after your changes when eviction is disabled? - did you compare performance eviction enabled vs eviction disabled? Thanks > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Labels: important > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955633#comment-15955633 ] ASF GitHub Bot commented on IGNITE-4534: GitHub user glukos opened a pull request: https://github.com/apache/ignite/pull/1731 IGNITE-4534, up-to-date with ignite-3477-master All changes from IGNITE-4534 applied over ignite-3477-master. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-4534-from-3477-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/1731.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1731 commit b127153b2d2733aab1881b056d2dd2bd7f374d81 Author: Sergey ChugunovDate: 2017-04-03T15:29:52Z Merge mistake was corrected, obsolete tests were removed commit 281efd4b9f0577d55260c7ea53c450e6182085c6 Author: Ilya Lantukh Date: 2017-04-03T16:19:30Z Fixed GridCacheRebalancingUnmarshallingFailedSelfTest. commit dce205fe288e048707596bb0b467fe562e93e845 Author: Ilya Lantukh Date: 2017-04-03T16:20:41Z Merge branch 'ignite-3477-master' of https://github.com/gridgain/apache-ignite into ignite-3477-master commit 98d92d89d69497172a6d33d88c17c1b05396ae04 Author: Ivan Rakov Date: 2017-04-04T18:53:35Z IGNITE-4534: Implement offheap eviction policies based on page memory > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Labels: important > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942992#comment-15942992 ] ASF GitHub Bot commented on IGNITE-4534: GitHub user glukos opened a pull request: https://github.com/apache/ignite/pull/1674 IGNITE-4534: Implement offheap eviction policies based on page memory Per-page eviction policies are implemented, algorithm (random-LRU or random-K-LRU) can be configured in MemoryPolicyConfiguration. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-4534 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/1674.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1674 commit 60ea031ebf19a7b6b8a205d762ae3b96ef2ed1ba Author: Ivan RakovDate: 2017-02-28T10:56:40Z IGNITE-4534: First approach to evict by page index commit c5d45a4fa575a444537db7a933a8f489574ccf83 Author: Ivan Rakov Date: 2017-02-28T11:35:36Z IGNITE-4534: First approach to evict by page index commit 27d024048bf6689e59548bfdf67461665b9ee06b Author: Ivan Rakov Date: 2017-03-02T17:34:19Z IGNITE-4534: Random-LRU implemented commit 0b7c770e4e12826e45a7716ec408129f68cf6ebb Author: Ivan Rakov Date: 2017-03-03T12:04:00Z IGNITE-4534: Bugs fixed. Works fine for small objects, found bug with big objects (size > pageSize) commit a8d74454f6e0c67632c5a2f2a10a41e8e2fcbe12 Author: Ivan Rakov Date: 2017-03-03T13:02:13Z IGNITE-4534: Smoke test added commit f5ff54a9945477768bf74161c8c51d4ce83d4b62 Author: Sergey Chugunov Date: 2017-03-02T16:24:20Z IGNITE-4758 multiple MemoryPolicies were implemented commit 5479f947f672e28fc39f92e8a489c016f96d10f0 Author: Sergey Chugunov Date: 2017-03-09T10:17:45Z Merge branch 'ignite-3477' into ignite-4758 commit db1087e28f438c494d96afea92de2b454f369c46 Author: Sergey Chugunov Date: 2017-03-09T12:13:17Z IGNITE-4758 added back 'freeList' parameter on RowStore constructor commit 4cd7cfab87f385fd6633f866d7f63625c010b649 Author: Ivan Rakov Date: 2017-03-13T12:17:24Z IGNITE-4534: Fixed bug with reading fragmented data row commit 26458a89bef395325232172dfcc20403c8d00eaa Author: Ivan Rakov Date: 2017-03-13T12:39:23Z Merge branch 'ignite-4758' into ignite-4534 # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/processors/cache/database/CacheDataRowAdapter.java # modules/core/src/main/java/org/apache/ignite/internal/processors/cache/database/IgniteCacheDatabaseSharedManager.java commit a0ae590e0411a1d029dfc515edf66fe45eb7635b Author: Ivan Rakov Date: 2017-03-13T15:00:22Z IGNITE-4578: Refactoring - configuration is now available from policy; memory policy map is introduced. commit df92f49048905a0245cf1daa0499cf96dd3c7238 Author: Ivan Rakov Date: 2017-03-13T15:01:13Z Merge branch 'ignite-4758' into ignite-4534 commit 9a5a4f17940ee902b27cd06998fb235553e4dea9 Author: Ivan Rakov Date: 2017-03-13T16:54:14Z IGNITE-4534: Integrated memory policy changes commit 9eaa00ef1a0d0124a23b59ef1245fa4e6df25478 Author: Sergey Chugunov Date: 2017-03-13T16:20:12Z IGNITE-4758 Indexing SPI tests fix, CPP tests configs refactoring, documentation improvements commit 89ce9b672223110b7f3b2ab36117772e9ddae06b Author: Sergey Chugunov Date: 2017-03-15T08:40:41Z IGNITE-4758 some tests were added, code was improved to avoid NPE on client nodes, documentation was detailed commit bf491b3a2fe94d05ff77d812809c02ce908036cf Author: Sergey Chugunov Date: 2017-03-16T11:16:51Z IGNITE-4758 NPE on metrics gathering on client node was fixed commit a4cbde0cfe25e9e40ba06882dc97b5d815cb4cf6 Author: Sergey Chugunov Date: 2017-03-20T14:27:53Z IGNITE-4758 bug with initializing default memory pool was fixed commit 1faea11b1e3f360b592cecd88149d86d8ba30e3a Author: Ivan Rakov Date: 2017-03-20T17:14:53Z Merge branch 'ignite-4758' into ignite-4534 # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/processors/cache/database/IgniteCacheDatabaseSharedManager.java commit df37f1f04d73a70ffd80c4bc6022a395ca7846ce Author: Ivan Rakov Date: 2017-03-20T17:16:52Z Fixing after merging ignite-4758 commit
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869822#comment-15869822 ] Alexey Goncharuk commented on IGNITE-4534: -- Alexey, In Ignite 1.x architecture there were 3 major areas where entries could be stored: on-heap, off-heap and swap. In Ignite 2.0 we are moving towards choosing off-heap as the primary storage and keeping some of the hot entries on-heap. However, in both Ignite 1.x and Ignite 2.x, using eviction implies that either a user can easily reconstruct evicted entries on-the-fly, or there is a backing write-through cache store keeping cache entries. Otherwise, eviction can effectively remove data from the cache. > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869661#comment-15869661 ] Ivan Rakov commented on IGNITE-4534: [~Alexey Kuznetsov], it's just another use case. This ticket covers "removing" eviction. If user needs "local disk" eviction, he can provide custom implementation of PageMemory. Another option is using cache in memory-mapped file mode, then OS will take care of swapping pages. > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869632#comment-15869632 ] Alexey Kuznetsov commented on IGNITE-4534: -- [~ivan.glukos] residing in off-heap makes jvm pauses shorter, ah, its reasonable feature. So user should back cache manually ? ah, it seems uncomfortable at first glance > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869614#comment-15869614 ] Ivan Rakov commented on IGNITE-4534: [~Alexey Kuznetsov], you're describing Ignite 1x layout. In 2.0 entries normally reside in off-heap and get removed upon eviction. We assume that if user configures eviction, he backs cache with external persistent database (or agrees with possible data loss). > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869596#comment-15869596 ] Alexey Kuznetsov commented on IGNITE-4534: -- [~glukos] im newby , i thought there are 2 memory areas where entries could reside : java heap memory, and permanent disk space. Eviction makes files go permanent disk space > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869583#comment-15869583 ] Ivan Rakov commented on IGNITE-4534: Have thoughts about re-implemeting them off-heap. Though, efficient LRU off-heap implementation is still a subject for research. > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869564#comment-15869564 ] Alexey Kuznetsov commented on IGNITE-4534: -- [~ivan.glukos] you have thoughts on removing on-heap policies ? > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869558#comment-15869558 ] Ivan Rakov commented on IGNITE-4534: Alexey, thanks! This will come in handy if we'll decide to keep on-heap eviction policies. > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory
[ https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869428#comment-15869428 ] Alexey Kuznetsov commented on IGNITE-4534: -- i've refactored eviction policies, so it might be helpful : https://github.com/voipp/ignite/pull/8/commits/48961834733d31bad6ec9ba9b642cea280ae04b3 > Implement offheap eviction policies based on page memory > > > Key: IGNITE-4534 > URL: https://issues.apache.org/jira/browse/IGNITE-4534 > Project: Ignite > Issue Type: Sub-task > Components: cache >Reporter: Alexey Goncharuk >Assignee: Ivan Rakov > Fix For: 2.0 > > > Since the internal structure of offheap storage has changed, we need to > re-implement Fifo, Lru and Sorted eviction policies -- This message was sent by Atlassian JIRA (v6.3.15#6346)