[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017184#comment-16017184
 ] 

ASF GitHub Bot commented on IGNITE-4534:


Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1758


> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Alexey Goncharuk
>  Labels: important
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961820#comment-15961820
 ] 

ASF GitHub Bot commented on IGNITE-4534:


Github user glukos closed the pull request at:

https://github.com/apache/ignite/pull/1731


> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
>  Labels: important
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961819#comment-15961819
 ] 

ASF GitHub Bot commented on IGNITE-4534:


Github user glukos closed the pull request at:

https://github.com/apache/ignite/pull/1674


> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
>  Labels: important
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[Semen Boikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956532#comment-15956532
 ] 

Semen Boikov commented on IGNITE-4534:
--

Ivan,

I'm reviewing your changes, meanwhile I want to know if you tested evictions 
impact on performance:
- did you compare performance before/after your changes when eviction is 
disabled?
- did you compare performance eviction enabled vs eviction disabled?

Thanks

> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
>  Labels: important
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955633#comment-15955633
 ] 

ASF GitHub Bot commented on IGNITE-4534:


GitHub user glukos opened a pull request:

https://github.com/apache/ignite/pull/1731

IGNITE-4534, up-to-date with ignite-3477-master

All changes from IGNITE-4534 applied over ignite-3477-master.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite 
ignite-4534-from-3477-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1731.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1731


commit b127153b2d2733aab1881b056d2dd2bd7f374d81
Author: Sergey Chugunov 
Date:   2017-04-03T15:29:52Z

Merge mistake was corrected, obsolete tests were removed

commit 281efd4b9f0577d55260c7ea53c450e6182085c6
Author: Ilya Lantukh 
Date:   2017-04-03T16:19:30Z

Fixed GridCacheRebalancingUnmarshallingFailedSelfTest.

commit dce205fe288e048707596bb0b467fe562e93e845
Author: Ilya Lantukh 
Date:   2017-04-03T16:20:41Z

Merge branch 'ignite-3477-master' of 
https://github.com/gridgain/apache-ignite into ignite-3477-master

commit 98d92d89d69497172a6d33d88c17c1b05396ae04
Author: Ivan Rakov 
Date:   2017-04-04T18:53:35Z

IGNITE-4534: Implement offheap eviction policies based on page memory




> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
>  Labels: important
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942992#comment-15942992
 ] 

ASF GitHub Bot commented on IGNITE-4534:


GitHub user glukos opened a pull request:

https://github.com/apache/ignite/pull/1674

IGNITE-4534: Implement offheap eviction policies based on page memory

Per-page eviction policies are implemented, algorithm (random-LRU or 
random-K-LRU) can be configured in MemoryPolicyConfiguration.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4534

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1674.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1674


commit 60ea031ebf19a7b6b8a205d762ae3b96ef2ed1ba
Author: Ivan Rakov 
Date:   2017-02-28T10:56:40Z

IGNITE-4534: First approach to evict by page index

commit c5d45a4fa575a444537db7a933a8f489574ccf83
Author: Ivan Rakov 
Date:   2017-02-28T11:35:36Z

IGNITE-4534: First approach to evict by page index

commit 27d024048bf6689e59548bfdf67461665b9ee06b
Author: Ivan Rakov 
Date:   2017-03-02T17:34:19Z

IGNITE-4534: Random-LRU implemented

commit 0b7c770e4e12826e45a7716ec408129f68cf6ebb
Author: Ivan Rakov 
Date:   2017-03-03T12:04:00Z

IGNITE-4534: Bugs fixed. Works fine for small objects, found bug with big 
objects (size > pageSize)

commit a8d74454f6e0c67632c5a2f2a10a41e8e2fcbe12
Author: Ivan Rakov 
Date:   2017-03-03T13:02:13Z

IGNITE-4534: Smoke test added

commit f5ff54a9945477768bf74161c8c51d4ce83d4b62
Author: Sergey Chugunov 
Date:   2017-03-02T16:24:20Z

IGNITE-4758 multiple MemoryPolicies were implemented

commit 5479f947f672e28fc39f92e8a489c016f96d10f0
Author: Sergey Chugunov 
Date:   2017-03-09T10:17:45Z

Merge branch 'ignite-3477' into ignite-4758

commit db1087e28f438c494d96afea92de2b454f369c46
Author: Sergey Chugunov 
Date:   2017-03-09T12:13:17Z

IGNITE-4758 added back 'freeList' parameter on RowStore constructor

commit 4cd7cfab87f385fd6633f866d7f63625c010b649
Author: Ivan Rakov 
Date:   2017-03-13T12:17:24Z

IGNITE-4534: Fixed bug with reading fragmented data row

commit 26458a89bef395325232172dfcc20403c8d00eaa
Author: Ivan Rakov 
Date:   2017-03-13T12:39:23Z

Merge branch 'ignite-4758' into ignite-4534

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/database/CacheDataRowAdapter.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/database/IgniteCacheDatabaseSharedManager.java

commit a0ae590e0411a1d029dfc515edf66fe45eb7635b
Author: Ivan Rakov 
Date:   2017-03-13T15:00:22Z

IGNITE-4578: Refactoring - configuration is now available from policy; 
memory policy map is introduced.

commit df92f49048905a0245cf1daa0499cf96dd3c7238
Author: Ivan Rakov 
Date:   2017-03-13T15:01:13Z

Merge branch 'ignite-4758' into ignite-4534

commit 9a5a4f17940ee902b27cd06998fb235553e4dea9
Author: Ivan Rakov 
Date:   2017-03-13T16:54:14Z

IGNITE-4534: Integrated memory policy changes

commit 9eaa00ef1a0d0124a23b59ef1245fa4e6df25478
Author: Sergey Chugunov 
Date:   2017-03-13T16:20:12Z

IGNITE-4758 Indexing SPI tests fix, CPP tests configs refactoring, 
documentation improvements

commit 89ce9b672223110b7f3b2ab36117772e9ddae06b
Author: Sergey Chugunov 
Date:   2017-03-15T08:40:41Z

IGNITE-4758 some tests were added, code was improved to avoid NPE on client 
nodes, documentation was detailed

commit bf491b3a2fe94d05ff77d812809c02ce908036cf
Author: Sergey Chugunov 
Date:   2017-03-16T11:16:51Z

IGNITE-4758 NPE on metrics gathering on client node was fixed

commit a4cbde0cfe25e9e40ba06882dc97b5d815cb4cf6
Author: Sergey Chugunov 
Date:   2017-03-20T14:27:53Z

IGNITE-4758 bug with initializing default memory pool was fixed

commit 1faea11b1e3f360b592cecd88149d86d8ba30e3a
Author: Ivan Rakov 
Date:   2017-03-20T17:14:53Z

Merge branch 'ignite-4758' into ignite-4534

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/database/IgniteCacheDatabaseSharedManager.java

commit df37f1f04d73a70ffd80c4bc6022a395ca7846ce
Author: Ivan Rakov 
Date:   2017-03-20T17:16:52Z

Fixing after merging ignite-4758

commit 

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[Alexey Goncharuk (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869822#comment-15869822
 ] 

Alexey Goncharuk commented on IGNITE-4534:
--

Alexey,

In Ignite 1.x architecture there were 3 major areas where entries could be 
stored: on-heap, off-heap and swap. In Ignite 2.0 we are moving towards 
choosing off-heap as the primary storage and keeping some of the hot entries 
on-heap.

However, in both Ignite 1.x and Ignite 2.x, using eviction implies that either 
a user can easily reconstruct evicted entries on-the-fly, or there is a backing 
write-through cache store keeping cache entries. Otherwise, eviction can 
effectively remove data from the cache.

> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[Ivan Rakov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869661#comment-15869661
 ] 

Ivan Rakov commented on IGNITE-4534:


[~Alexey Kuznetsov], it's just another use case. This ticket covers "removing" 
eviction. 
If user needs "local disk" eviction, he can provide custom implementation of 
PageMemory. Another option is using cache in memory-mapped file mode, then OS 
will take care of swapping pages.

> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[Alexey Kuznetsov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869632#comment-15869632
 ] 

Alexey Kuznetsov commented on IGNITE-4534:
--

[~ivan.glukos] residing in off-heap makes jvm pauses shorter, ah, its 
reasonable feature. So user should back cache manually ? ah, it seems 
uncomfortable at first glance

> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[Ivan Rakov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869614#comment-15869614
 ] 

Ivan Rakov commented on IGNITE-4534:


[~Alexey Kuznetsov], you're describing Ignite 1x layout. 
In 2.0 entries normally reside in off-heap and get removed upon eviction. We 
assume that if user configures eviction, he backs cache with external 
persistent database (or agrees with possible data loss).

> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[Alexey Kuznetsov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869596#comment-15869596
 ] 

Alexey Kuznetsov commented on IGNITE-4534:
--

[~glukos] im newby , i thought there are 2 memory areas where entries could 
reside : java heap memory, and permanent disk space. Eviction makes files go 
permanent disk space

> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[Ivan Rakov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869583#comment-15869583
 ] 

Ivan Rakov commented on IGNITE-4534:


Have thoughts about re-implemeting them off-heap. Though, efficient LRU 
off-heap implementation is still a subject for research.

> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[Alexey Kuznetsov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869564#comment-15869564
 ] 

Alexey Kuznetsov commented on IGNITE-4534:
--

[~ivan.glukos] you have thoughts on removing on-heap policies ?

> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[Ivan Rakov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869558#comment-15869558
 ] 

Ivan Rakov commented on IGNITE-4534:


Alexey, thanks! This will come in handy if we'll decide to keep on-heap 
eviction policies.

> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4534) Implement offheap eviction policies based on page memory

[Alexey Kuznetsov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869428#comment-15869428
 ] 

Alexey Kuznetsov commented on IGNITE-4534:
--

i've refactored eviction policies, so it might be helpful : 
https://github.com/voipp/ignite/pull/8/commits/48961834733d31bad6ec9ba9b642cea280ae04b3

> Implement offheap eviction policies based on page memory
> 
>
> Key: IGNITE-4534
> URL: https://issues.apache.org/jira/browse/IGNITE-4534
> Project: Ignite
>  Issue Type: Sub-task
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Ivan Rakov
> Fix For: 2.0
>
>
> Since the internal structure of offheap storage has changed, we need to 
> re-implement Fifo, Lru and Sorted eviction policies



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)