Re: Eviction policies with persistence

[Denis Magda]

Thanks folks. I’ll improve the doc by 2.4 release:
IGNITE-7645 - Clarify eviction policy documentation 


—
Denis

> On Feb 6, 2018, at 11:25 PM, Alexey Goncharuk  
> wrote:
> 
> Denis,
> 
> Yes, if there are no concurrent updates and there are no ongoing
> transactions preventing the entries to be evicted, then the whole page will
> be purged.
> 
> 2018-02-07 0:23 GMT+03:00 Denis Magda :
> 
>> Alexey,
>> 
>> Let me understand this part because it’s still not crystal clear to me:
>> 
>>> Now as per the
>>> eviction mechanism, it is both per-page and per-entry: first, we choose a
>>> page which fits most for eviction, however, we cannot simply erase the
>> page
>>> because quite a lot of data structures are referring to that page. To
>> deal
>>> with it, we read keys that the chosen page contains and then clear
>>> individual cache entries, which in turn will clear up all necessary
>> links.
>> 
>> 
>> However will all the keys-values from a chosen page be removed eventually
>> during the eviction phase? If it’s true then it’s still a page based
>> eviction - we select 5 random oldest pages and purge all the key-values
>> from them.
>> 
>> —
>> Denis
>> 
>>> On Feb 4, 2018, at 12:05 PM, Alexey Goncharuk <
>> alexey.goncha...@gmail.com> wrote:
>>> 
>>> Guys,
>>> 
>>> To clarify the questions, I would like to reiterate over the mechanics of
>>> evictions and then suggest a renaming that should resolve such things in
>>> the future.
>>> 
>>> First of all, Lucas is right - eviction policy only makes sense when
>> native
>>> persistence is disabled because what it does is actually freeing up
>> memory
>>> when a user hits the memory limit. The only way to do this is to destroy
>>> inserted data because there is no other way to free memory. Now as per
>> the
>>> eviction mechanism, it is both per-page and per-entry: first, we choose a
>>> page which fits most for eviction, however, we cannot simply erase the
>> page
>>> because quite a lot of data structures are referring to that page. To
>> deal
>>> with it, we read keys that the chosen page contains and then clear
>>> individual cache entries, which in turn will clear up all necessary
>> links.
>>> If there are no concurrent updates, the page becomes empty and will be
>>> reused for other user data. This is exactly what is explained on the wiki
>>> page (at least in my reading of the page).
>>> 
>>> Second, at this point, I would rename page management mechanism with
>>> enabled persistence from 'eviction' to 'page replacement' or 'page
>>> swapping', because it does not destroy any data, but rather replaces a
>>> chosen buffer in memory from one page to another. The content of neither
>>> pages does not change during page replacement. This mechanism is unlikely
>>> to be selected by a user because the effectiveness of page replacements
>>> heavily depends on internal data structures and may change from version
>> to
>>> version, and may even be adaptive depending on the load pattern.
>>> 
>>> Hope this resolves the confusion.
>>> 
>>> --AG
>>> 
>>> 2018-02-03 1:03 GMT+03:00 Denis Magda :
>>> 
 Dmitriy,
 
 I’m surprised to hear that Random-LRU works at the entry level when
>> Ignite
 persistence is off. Frankly, this section confuses a lot:
 https://cwiki.apache.org/confluence/display/IGNITE/
 Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
 underthehood-Entryeviction 
 
 While it was assumed that the entry-based eviction works only for
>> on-heap
 caches:
 https://apacheignite.readme.io/docs/evictions <
 https://apacheignite.readme.io/docs/evictions>
 
 Alex G., please chime in and clarify the misunderstanding.
 
 —
 Denis
 
> On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov 
>> wrote:
> 
> Hi Val,
> 
> I think it is quite accurate because eviction in case PDS enabled or
>> not
> has quite different purposes.
> 
> 1. Let us consider PDS enabled and page eviction occurs. First of all
>> it
 is
> page based eviction, but not entry-based. Actually data is not removed,
 but
> only written to disk. We can address this page later by ID.
> PDS eviction is primarily the replacement of pages from memory to page
>> on
> disk. This eviction policy should ensure the maximum performance of the
> solution in the future. There is no data removal from grid in this
>> case.
> And Ignite does not allow users to configure the policy. If benchmarks
 show
> that a change in policy results in increased performance, then we can
> switch policy.
> 
> 2. Entry-based eviction (if there is no persistence is available) is
> slithly 

Re: Eviction policies with persistence

[Alexey Goncharuk]

Denis,

Yes, if there are no concurrent updates and there are no ongoing
transactions preventing the entries to be evicted, then the whole page will
be purged.

2018-02-07 0:23 GMT+03:00 Denis Magda :

> Alexey,
>
> Let me understand this part because it’s still not crystal clear to me:
>
> > Now as per the
> > eviction mechanism, it is both per-page and per-entry: first, we choose a
> > page which fits most for eviction, however, we cannot simply erase the
> page
> > because quite a lot of data structures are referring to that page. To
> deal
> > with it, we read keys that the chosen page contains and then clear
> > individual cache entries, which in turn will clear up all necessary
> links.
>
>
> However will all the keys-values from a chosen page be removed eventually
> during the eviction phase? If it’s true then it’s still a page based
> eviction - we select 5 random oldest pages and purge all the key-values
> from them.
>
> —
> Denis
>
> > On Feb 4, 2018, at 12:05 PM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
> >
> > Guys,
> >
> > To clarify the questions, I would like to reiterate over the mechanics of
> > evictions and then suggest a renaming that should resolve such things in
> > the future.
> >
> > First of all, Lucas is right - eviction policy only makes sense when
> native
> > persistence is disabled because what it does is actually freeing up
> memory
> > when a user hits the memory limit. The only way to do this is to destroy
> > inserted data because there is no other way to free memory. Now as per
> the
> > eviction mechanism, it is both per-page and per-entry: first, we choose a
> > page which fits most for eviction, however, we cannot simply erase the
> page
> > because quite a lot of data structures are referring to that page. To
> deal
> > with it, we read keys that the chosen page contains and then clear
> > individual cache entries, which in turn will clear up all necessary
> links.
> > If there are no concurrent updates, the page becomes empty and will be
> > reused for other user data. This is exactly what is explained on the wiki
> > page (at least in my reading of the page).
> >
> > Second, at this point, I would rename page management mechanism with
> > enabled persistence from 'eviction' to 'page replacement' or 'page
> > swapping', because it does not destroy any data, but rather replaces a
> > chosen buffer in memory from one page to another. The content of neither
> > pages does not change during page replacement. This mechanism is unlikely
> > to be selected by a user because the effectiveness of page replacements
> > heavily depends on internal data structures and may change from version
> to
> > version, and may even be adaptive depending on the load pattern.
> >
> > Hope this resolves the confusion.
> >
> > --AG
> >
> > 2018-02-03 1:03 GMT+03:00 Denis Magda :
> >
> >> Dmitriy,
> >>
> >> I’m surprised to hear that Random-LRU works at the entry level when
> Ignite
> >> persistence is off. Frankly, this section confuses a lot:
> >> https://cwiki.apache.org/confluence/display/IGNITE/
> >> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> >> underthehood-Entryeviction  >> confluence/display/IGNITE/Ignite+Durable+Memory+-+under+
> >> the+hood#IgniteDurableMemory-underthehood-Entryeviction>
> >>
> >> While it was assumed that the entry-based eviction works only for
> on-heap
> >> caches:
> >> https://apacheignite.readme.io/docs/evictions <
> >> https://apacheignite.readme.io/docs/evictions>
> >>
> >> Alex G., please chime in and clarify the misunderstanding.
> >>
> >> —
> >> Denis
> >>
> >>> On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov 
> wrote:
> >>>
> >>> Hi Val,
> >>>
> >>> I think it is quite accurate because eviction in case PDS enabled or
> not
> >>> has quite different purposes.
> >>>
> >>> 1. Let us consider PDS enabled and page eviction occurs. First of all
> it
> >> is
> >>> page based eviction, but not entry-based. Actually data is not removed,
> >> but
> >>> only written to disk. We can address this page later by ID.
> >>> PDS eviction is primarily the replacement of pages from memory to page
> on
> >>> disk. This eviction policy should ensure the maximum performance of the
> >>> solution in the future. There is no data removal from grid in this
> case.
> >>> And Ignite does not allow users to configure the policy. If benchmarks
> >> show
> >>> that a change in policy results in increased performance, then we can
> >>> switch policy.
> >>>
> >>> 2. Entry-based eviction (if there is no persistence is available) is
> >>> slithly different. User will need to reload data into grid in case
> entry
> >> is
> >>> evicted but required in cache. In that case Ignite provides selection
> of
> >>> policies.
> >>>
> >>> Sincerely,
> >>> Dmirtiy Pavlov
> >>>
> >>>
> >>>
> >>> чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
> >>> valentin.kuliche...@gmail.com>:
> >>>

Re: Eviction policies with persistence

[Valentin Kulichenko]

Guys, thanks for all the clarification, it becomes more clear now. I second
Denis' question though.

And I actually don't think that we need to rename anything here. The
concept of eviction as "freeing up memory when a user hits the memory
limit" makes sense to me with both persistence enabled and disabled, I
would not overcomplicate it. What we need to do os properly describe it.

The confusion is caused only by the way documentation is written, in
particular these two points:
1. We claim that "Configured eviction policy has no effect if Ignite
persistence is enabled" which to me sounds like there is no eviction in
this case at all. So it's completely unclear how memory is freed up when
persistence is used and what kind of control user has.
2. We claim that we have "page-based eviction for the off-heap memory data
sets" and "entry-based eviction for the optional on-heap cache" which based
on this discussion just doesn't seem to be true.

I think we need to have three separate sections here: one for off-heap with
persistence, another for off-heap without persistence, and the last one for
on-heap cache. Each section should provide detailed explanation of how
memory is cleaned up in this particular case and what options user has to
configure.

In addition, it makes sense to have warnings in case configuration is used
incorrectly. E.g. if pageEvictionMode is configured with persistence, or if
evictionPolicy is configured without on-heap cache.

-Val

On Tue, Feb 6, 2018 at 1:23 PM, Denis Magda  wrote:

> Alexey,
>
> Let me understand this part because it’s still not crystal clear to me:
>
> > Now as per the
> > eviction mechanism, it is both per-page and per-entry: first, we choose a
> > page which fits most for eviction, however, we cannot simply erase the
> page
> > because quite a lot of data structures are referring to that page. To
> deal
> > with it, we read keys that the chosen page contains and then clear
> > individual cache entries, which in turn will clear up all necessary
> links.
>
>
> However will all the keys-values from a chosen page be removed eventually
> during the eviction phase? If it’s true then it’s still a page based
> eviction - we select 5 random oldest pages and purge all the key-values
> from them.
>
> —
> Denis
>
> > On Feb 4, 2018, at 12:05 PM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
> >
> > Guys,
> >
> > To clarify the questions, I would like to reiterate over the mechanics of
> > evictions and then suggest a renaming that should resolve such things in
> > the future.
> >
> > First of all, Lucas is right - eviction policy only makes sense when
> native
> > persistence is disabled because what it does is actually freeing up
> memory
> > when a user hits the memory limit. The only way to do this is to destroy
> > inserted data because there is no other way to free memory. Now as per
> the
> > eviction mechanism, it is both per-page and per-entry: first, we choose a
> > page which fits most for eviction, however, we cannot simply erase the
> page
> > because quite a lot of data structures are referring to that page. To
> deal
> > with it, we read keys that the chosen page contains and then clear
> > individual cache entries, which in turn will clear up all necessary
> links.
> > If there are no concurrent updates, the page becomes empty and will be
> > reused for other user data. This is exactly what is explained on the wiki
> > page (at least in my reading of the page).
> >
> > Second, at this point, I would rename page management mechanism with
> > enabled persistence from 'eviction' to 'page replacement' or 'page
> > swapping', because it does not destroy any data, but rather replaces a
> > chosen buffer in memory from one page to another. The content of neither
> > pages does not change during page replacement. This mechanism is unlikely
> > to be selected by a user because the effectiveness of page replacements
> > heavily depends on internal data structures and may change from version
> to
> > version, and may even be adaptive depending on the load pattern.
> >
> > Hope this resolves the confusion.
> >
> > --AG
> >
> > 2018-02-03 1:03 GMT+03:00 Denis Magda :
> >
> >> Dmitriy,
> >>
> >> I’m surprised to hear that Random-LRU works at the entry level when
> Ignite
> >> persistence is off. Frankly, this section confuses a lot:
> >> https://cwiki.apache.org/confluence/display/IGNITE/
> >> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> >> underthehood-Entryeviction  >> confluence/display/IGNITE/Ignite+Durable+Memory+-+under+
> >> the+hood#IgniteDurableMemory-underthehood-Entryeviction>
> >>
> >> While it was assumed that the entry-based eviction works only for
> on-heap
> >> caches:
> >> https://apacheignite.readme.io/docs/evictions <
> >> https://apacheignite.readme.io/docs/evictions>
> >>
> >> Alex G., please chime in and clarify the misunderstanding.
> >>
> >> —
> >> Denis
> >>
> 

Re: Eviction policies with persistence

[Denis Magda]

Alexey,

Let me understand this part because it’s still not crystal clear to me:

> Now as per the
> eviction mechanism, it is both per-page and per-entry: first, we choose a
> page which fits most for eviction, however, we cannot simply erase the page
> because quite a lot of data structures are referring to that page. To deal
> with it, we read keys that the chosen page contains and then clear
> individual cache entries, which in turn will clear up all necessary links.


However will all the keys-values from a chosen page be removed eventually 
during the eviction phase? If it’s true then it’s still a page based eviction - 
we select 5 random oldest pages and purge all the key-values from them.

—
Denis 

> On Feb 4, 2018, at 12:05 PM, Alexey Goncharuk  
> wrote:
> 
> Guys,
> 
> To clarify the questions, I would like to reiterate over the mechanics of
> evictions and then suggest a renaming that should resolve such things in
> the future.
> 
> First of all, Lucas is right - eviction policy only makes sense when native
> persistence is disabled because what it does is actually freeing up memory
> when a user hits the memory limit. The only way to do this is to destroy
> inserted data because there is no other way to free memory. Now as per the
> eviction mechanism, it is both per-page and per-entry: first, we choose a
> page which fits most for eviction, however, we cannot simply erase the page
> because quite a lot of data structures are referring to that page. To deal
> with it, we read keys that the chosen page contains and then clear
> individual cache entries, which in turn will clear up all necessary links.
> If there are no concurrent updates, the page becomes empty and will be
> reused for other user data. This is exactly what is explained on the wiki
> page (at least in my reading of the page).
> 
> Second, at this point, I would rename page management mechanism with
> enabled persistence from 'eviction' to 'page replacement' or 'page
> swapping', because it does not destroy any data, but rather replaces a
> chosen buffer in memory from one page to another. The content of neither
> pages does not change during page replacement. This mechanism is unlikely
> to be selected by a user because the effectiveness of page replacements
> heavily depends on internal data structures and may change from version to
> version, and may even be adaptive depending on the load pattern.
> 
> Hope this resolves the confusion.
> 
> --AG
> 
> 2018-02-03 1:03 GMT+03:00 Denis Magda :
> 
>> Dmitriy,
>> 
>> I’m surprised to hear that Random-LRU works at the entry level when Ignite
>> persistence is off. Frankly, this section confuses a lot:
>> https://cwiki.apache.org/confluence/display/IGNITE/
>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
>> underthehood-Entryeviction > confluence/display/IGNITE/Ignite+Durable+Memory+-+under+
>> the+hood#IgniteDurableMemory-underthehood-Entryeviction>
>> 
>> While it was assumed that the entry-based eviction works only for on-heap
>> caches:
>> https://apacheignite.readme.io/docs/evictions <
>> https://apacheignite.readme.io/docs/evictions>
>> 
>> Alex G., please chime in and clarify the misunderstanding.
>> 
>> —
>> Denis
>> 
>>> On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov  wrote:
>>> 
>>> Hi Val,
>>> 
>>> I think it is quite accurate because eviction in case PDS enabled or not
>>> has quite different purposes.
>>> 
>>> 1. Let us consider PDS enabled and page eviction occurs. First of all it
>> is
>>> page based eviction, but not entry-based. Actually data is not removed,
>> but
>>> only written to disk. We can address this page later by ID.
>>> PDS eviction is primarily the replacement of pages from memory to page on
>>> disk. This eviction policy should ensure the maximum performance of the
>>> solution in the future. There is no data removal from grid in this case.
>>> And Ignite does not allow users to configure the policy. If benchmarks
>> show
>>> that a change in policy results in increased performance, then we can
>>> switch policy.
>>> 
>>> 2. Entry-based eviction (if there is no persistence is available) is
>>> slithly different. User will need to reload data into grid in case entry
>> is
>>> evicted but required in cache. In that case Ignite provides selection of
>>> policies.
>>> 
>>> Sincerely,
>>> Dmirtiy Pavlov
>>> 
>>> 
>>> 
>>> чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
>>> valentin.kuliche...@gmail.com>:
>>> 
 Guys,
 
 Thanks for responses. But I still fail to understand how it affects a
>> user.
 
 Are you saying that with persistence enabled Random-LRU is always used
 regardless of what is specified in pageEvictionMode, while in in-memory
 only scenario we can also utilize Random-2-LRU for eviction? Is this
 accurate? Are there any other differences?
 
 -Val
 
 On Thu, Feb 1, 2018 at 5:01 AM, Dmitry 

Re: Eviction policies with persistence

[Dmitry Pavlov]

Hi Folks,

Thank you for your replies! I've updated wiki according Lucas & Alexey's
notes. Thank you!

https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Pagereplacement(rotationwithdisk)

I've named this section 'Page replacement (rotation with disk)'. I'm going
to update also class naming in my current PR (3469
)

Sincerely,
Dmitriy Pavlov

пн, 5 февр. 2018 г. в 15:25, Ivan Rakov :

> Guys,
>
> +1 for renaming "eviction" for PDS mode.
> I'd choose "page replacement" over "page swapping" for two reasons:
> 1) We already have "replace" metrics in DataRegionMetrics. The less we
> change existing namings, the better.
> 2) "Page swapping" term is common in operating systems, and that's why
> we'll confuse our "page swapping" (which is essential of Ignite native
> persistence) with actual OS page swapping (which usually signals about
> ineffective hardware resources usage and is better to be avoided) in
> many discussions later.
>
> Best Regards,
> Ivan Rakov
>
> On 04.02.2018 23:42, Dmitry Pavlov wrote:
> > Hi Alex,
> >
> > Thank you for explanation. Do you mind if I copy this explanation to wiki
> > page?
> >
> > Strongly +1 for new term for page replacement mechanism for PDS mode. I
> > like 'page swapping' because it has analogue in operating systems.
> >
> > New term will also identify feature by class name PageEviction... and
> > PageSwap... will point to class purpose.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вс, 4 февр. 2018 г. в 23:05, Alexey Goncharuk <
> alexey.goncha...@gmail.com>:
> >
> >> Guys,
> >>
> >> To clarify the questions, I would like to reiterate over the mechanics
> of
> >> evictions and then suggest a renaming that should resolve such things in
> >> the future.
> >>
> >> First of all, Lucas is right - eviction policy only makes sense when
> native
> >> persistence is disabled because what it does is actually freeing up
> memory
> >> when a user hits the memory limit. The only way to do this is to destroy
> >> inserted data because there is no other way to free memory. Now as per
> the
> >> eviction mechanism, it is both per-page and per-entry: first, we choose
> a
> >> page which fits most for eviction, however, we cannot simply erase the
> page
> >> because quite a lot of data structures are referring to that page. To
> deal
> >> with it, we read keys that the chosen page contains and then clear
> >> individual cache entries, which in turn will clear up all necessary
> links.
> >> If there are no concurrent updates, the page becomes empty and will be
> >> reused for other user data. This is exactly what is explained on the
> wiki
> >> page (at least in my reading of the page).
> >>
> >> Second, at this point, I would rename page management mechanism with
> >> enabled persistence from 'eviction' to 'page replacement' or 'page
> >> swapping', because it does not destroy any data, but rather replaces a
> >> chosen buffer in memory from one page to another. The content of neither
> >> pages does not change during page replacement. This mechanism is
> unlikely
> >> to be selected by a user because the effectiveness of page replacements
> >> heavily depends on internal data structures and may change from version
> to
> >> version, and may even be adaptive depending on the load pattern.
> >>
> >> Hope this resolves the confusion.
> >>
> >> --AG
> >>
> >> 2018-02-03 1:03 GMT+03:00 Denis Magda :
> >>
> >>> Dmitriy,
> >>>
> >>> I’m surprised to hear that Random-LRU works at the entry level when
> >> Ignite
> >>> persistence is off. Frankly, this section confuses a lot:
> >>> https://cwiki.apache.org/confluence/display/IGNITE/
> >>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> >>> underthehood-Entryeviction  >>> confluence/display/IGNITE/Ignite+Durable+Memory+-+under+
> >>> the+hood#IgniteDurableMemory-underthehood-Entryeviction>
> >>>
> >>> While it was assumed that the entry-based eviction works only for
> on-heap
> >>> caches:
> >>> https://apacheignite.readme.io/docs/evictions <
> >>> https://apacheignite.readme.io/docs/evictions>
> >>>
> >>> Alex G., please chime in and clarify the misunderstanding.
> >>>
> >>> —
> >>> Denis
> >>>
>  On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov 
> >> wrote:
>  Hi Val,
> 
>  I think it is quite accurate because eviction in case PDS enabled or
> >> not
>  has quite different purposes.
> 
>  1. Let us consider PDS enabled and page eviction occurs. First of all
> >> it
> >>> is
>  page based eviction, but not entry-based. Actually data is not
> removed,
> >>> but
>  only written to disk. We can address this page later by ID.
>  PDS eviction is primarily the replacement of pages from memory to page
> >> on
>  disk. This eviction policy should ensure the maximum performance of
> the
>  

Re: Eviction policies with persistence

[Ivan Rakov]

Guys,

+1 for renaming "eviction" for PDS mode.
I'd choose "page replacement" over "page swapping" for two reasons:
1) We already have "replace" metrics in DataRegionMetrics. The less we 
change existing namings, the better.
2) "Page swapping" term is common in operating systems, and that's why 
we'll confuse our "page swapping" (which is essential of Ignite native 
persistence) with actual OS page swapping (which usually signals about 
ineffective hardware resources usage and is better to be avoided) in 
many discussions later.


Best Regards,
Ivan Rakov

On 04.02.2018 23:42, Dmitry Pavlov wrote:

Hi Alex,

Thank you for explanation. Do you mind if I copy this explanation to wiki
page?

Strongly +1 for new term for page replacement mechanism for PDS mode. I
like 'page swapping' because it has analogue in operating systems.

New term will also identify feature by class name PageEviction... and
PageSwap... will point to class purpose.

Sincerely,
Dmitriy Pavlov

вс, 4 февр. 2018 г. в 23:05, Alexey Goncharuk :


Guys,

To clarify the questions, I would like to reiterate over the mechanics of
evictions and then suggest a renaming that should resolve such things in
the future.

First of all, Lucas is right - eviction policy only makes sense when native
persistence is disabled because what it does is actually freeing up memory
when a user hits the memory limit. The only way to do this is to destroy
inserted data because there is no other way to free memory. Now as per the
eviction mechanism, it is both per-page and per-entry: first, we choose a
page which fits most for eviction, however, we cannot simply erase the page
because quite a lot of data structures are referring to that page. To deal
with it, we read keys that the chosen page contains and then clear
individual cache entries, which in turn will clear up all necessary links.
If there are no concurrent updates, the page becomes empty and will be
reused for other user data. This is exactly what is explained on the wiki
page (at least in my reading of the page).

Second, at this point, I would rename page management mechanism with
enabled persistence from 'eviction' to 'page replacement' or 'page
swapping', because it does not destroy any data, but rather replaces a
chosen buffer in memory from one page to another. The content of neither
pages does not change during page replacement. This mechanism is unlikely
to be selected by a user because the effectiveness of page replacements
heavily depends on internal data structures and may change from version to
version, and may even be adaptive depending on the load pattern.

Hope this resolves the confusion.

--AG

2018-02-03 1:03 GMT+03:00 Denis Magda :


Dmitriy,

I’m surprised to hear that Random-LRU works at the entry level when

Ignite

persistence is off. Frankly, this section confuses a lot:
https://cwiki.apache.org/confluence/display/IGNITE/
Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
underthehood-Entryeviction 

While it was assumed that the entry-based eviction works only for on-heap
caches:
https://apacheignite.readme.io/docs/evictions <
https://apacheignite.readme.io/docs/evictions>

Alex G., please chime in and clarify the misunderstanding.

—
Denis


On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov 

wrote:

Hi Val,

I think it is quite accurate because eviction in case PDS enabled or

not

has quite different purposes.

1. Let us consider PDS enabled and page eviction occurs. First of all

it

is

page based eviction, but not entry-based. Actually data is not removed,

but

only written to disk. We can address this page later by ID.
PDS eviction is primarily the replacement of pages from memory to page

on

disk. This eviction policy should ensure the maximum performance of the
solution in the future. There is no data removal from grid in this

case.

And Ignite does not allow users to configure the policy. If benchmarks

show

that a change in policy results in increased performance, then we can
switch policy.

2. Entry-based eviction (if there is no persistence is available) is
slithly different. User will need to reload data into grid in case

entry

is

evicted but required in cache. In that case Ignite provides selection

of

policies.

Sincerely,
Dmirtiy Pavlov



чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:


Guys,

Thanks for responses. But I still fail to understand how it affects a

user.

Are you saying that with persistence enabled Random-LRU is always used
regardless of what is specified in pageEvictionMode, while in

in-memory

only scenario we can also utilize Random-2-LRU for eviction? Is this
accurate? Are there any other differences?

-Val

On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov 
wrote:


Hi 

Re: Eviction policies with persistence

[Dmitry Pavlov]

Hi Alex,

Thank you for explanation. Do you mind if I copy this explanation to wiki
page?

Strongly +1 for new term for page replacement mechanism for PDS mode. I
like 'page swapping' because it has analogue in operating systems.

New term will also identify feature by class name PageEviction... and
PageSwap... will point to class purpose.

Sincerely,
Dmitriy Pavlov

вс, 4 февр. 2018 г. в 23:05, Alexey Goncharuk :

> Guys,
>
> To clarify the questions, I would like to reiterate over the mechanics of
> evictions and then suggest a renaming that should resolve such things in
> the future.
>
> First of all, Lucas is right - eviction policy only makes sense when native
> persistence is disabled because what it does is actually freeing up memory
> when a user hits the memory limit. The only way to do this is to destroy
> inserted data because there is no other way to free memory. Now as per the
> eviction mechanism, it is both per-page and per-entry: first, we choose a
> page which fits most for eviction, however, we cannot simply erase the page
> because quite a lot of data structures are referring to that page. To deal
> with it, we read keys that the chosen page contains and then clear
> individual cache entries, which in turn will clear up all necessary links.
> If there are no concurrent updates, the page becomes empty and will be
> reused for other user data. This is exactly what is explained on the wiki
> page (at least in my reading of the page).
>
> Second, at this point, I would rename page management mechanism with
> enabled persistence from 'eviction' to 'page replacement' or 'page
> swapping', because it does not destroy any data, but rather replaces a
> chosen buffer in memory from one page to another. The content of neither
> pages does not change during page replacement. This mechanism is unlikely
> to be selected by a user because the effectiveness of page replacements
> heavily depends on internal data structures and may change from version to
> version, and may even be adaptive depending on the load pattern.
>
> Hope this resolves the confusion.
>
> --AG
>
> 2018-02-03 1:03 GMT+03:00 Denis Magda :
>
> > Dmitriy,
> >
> > I’m surprised to hear that Random-LRU works at the entry level when
> Ignite
> > persistence is off. Frankly, this section confuses a lot:
> > https://cwiki.apache.org/confluence/display/IGNITE/
> > Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> > underthehood-Entryeviction  > confluence/display/IGNITE/Ignite+Durable+Memory+-+under+
> > the+hood#IgniteDurableMemory-underthehood-Entryeviction>
> >
> > While it was assumed that the entry-based eviction works only for on-heap
> > caches:
> > https://apacheignite.readme.io/docs/evictions <
> > https://apacheignite.readme.io/docs/evictions>
> >
> > Alex G., please chime in and clarify the misunderstanding.
> >
> > —
> > Denis
> >
> > > On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov 
> wrote:
> > >
> > > Hi Val,
> > >
> > > I think it is quite accurate because eviction in case PDS enabled or
> not
> > > has quite different purposes.
> > >
> > > 1. Let us consider PDS enabled and page eviction occurs. First of all
> it
> > is
> > > page based eviction, but not entry-based. Actually data is not removed,
> > but
> > > only written to disk. We can address this page later by ID.
> > > PDS eviction is primarily the replacement of pages from memory to page
> on
> > > disk. This eviction policy should ensure the maximum performance of the
> > > solution in the future. There is no data removal from grid in this
> case.
> > > And Ignite does not allow users to configure the policy. If benchmarks
> > show
> > > that a change in policy results in increased performance, then we can
> > > switch policy.
> > >
> > > 2. Entry-based eviction (if there is no persistence is available) is
> > > slithly different. User will need to reload data into grid in case
> entry
> > is
> > > evicted but required in cache. In that case Ignite provides selection
> of
> > > policies.
> > >
> > > Sincerely,
> > > Dmirtiy Pavlov
> > >
> > >
> > >
> > > чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com>:
> > >
> > >> Guys,
> > >>
> > >> Thanks for responses. But I still fail to understand how it affects a
> > user.
> > >>
> > >> Are you saying that with persistence enabled Random-LRU is always used
> > >> regardless of what is specified in pageEvictionMode, while in
> in-memory
> > >> only scenario we can also utilize Random-2-LRU for eviction? Is this
> > >> accurate? Are there any other differences?
> > >>
> > >> -Val
> > >>
> > >> On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov 
> > >> wrote:
> > >>
> > >>> Hi Vyacheslav,
> > >>>
> > >>> Yes, this is right, but for now PDS page based eviction uses
> > Random-LRU,
> > >>> not Random-2-LRU. Something may be changed in future Ignite releases,
> > but
> > 

Re: Eviction policies with persistence

[Alexey Goncharuk]

Guys,

To clarify the questions, I would like to reiterate over the mechanics of
evictions and then suggest a renaming that should resolve such things in
the future.

First of all, Lucas is right - eviction policy only makes sense when native
persistence is disabled because what it does is actually freeing up memory
when a user hits the memory limit. The only way to do this is to destroy
inserted data because there is no other way to free memory. Now as per the
eviction mechanism, it is both per-page and per-entry: first, we choose a
page which fits most for eviction, however, we cannot simply erase the page
because quite a lot of data structures are referring to that page. To deal
with it, we read keys that the chosen page contains and then clear
individual cache entries, which in turn will clear up all necessary links.
If there are no concurrent updates, the page becomes empty and will be
reused for other user data. This is exactly what is explained on the wiki
page (at least in my reading of the page).

Second, at this point, I would rename page management mechanism with
enabled persistence from 'eviction' to 'page replacement' or 'page
swapping', because it does not destroy any data, but rather replaces a
chosen buffer in memory from one page to another. The content of neither
pages does not change during page replacement. This mechanism is unlikely
to be selected by a user because the effectiveness of page replacements
heavily depends on internal data structures and may change from version to
version, and may even be adaptive depending on the load pattern.

Hope this resolves the confusion.

--AG

2018-02-03 1:03 GMT+03:00 Denis Magda :

> Dmitriy,
>
> I’m surprised to hear that Random-LRU works at the entry level when Ignite
> persistence is off. Frankly, this section confuses a lot:
> https://cwiki.apache.org/confluence/display/IGNITE/
> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> underthehood-Entryeviction  confluence/display/IGNITE/Ignite+Durable+Memory+-+under+
> the+hood#IgniteDurableMemory-underthehood-Entryeviction>
>
> While it was assumed that the entry-based eviction works only for on-heap
> caches:
> https://apacheignite.readme.io/docs/evictions <
> https://apacheignite.readme.io/docs/evictions>
>
> Alex G., please chime in and clarify the misunderstanding.
>
> —
> Denis
>
> > On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov  wrote:
> >
> > Hi Val,
> >
> > I think it is quite accurate because eviction in case PDS enabled or not
> > has quite different purposes.
> >
> > 1. Let us consider PDS enabled and page eviction occurs. First of all it
> is
> > page based eviction, but not entry-based. Actually data is not removed,
> but
> > only written to disk. We can address this page later by ID.
> > PDS eviction is primarily the replacement of pages from memory to page on
> > disk. This eviction policy should ensure the maximum performance of the
> > solution in the future. There is no data removal from grid in this case.
> > And Ignite does not allow users to configure the policy. If benchmarks
> show
> > that a change in policy results in increased performance, then we can
> > switch policy.
> >
> > 2. Entry-based eviction (if there is no persistence is available) is
> > slithly different. User will need to reload data into grid in case entry
> is
> > evicted but required in cache. In that case Ignite provides selection of
> > policies.
> >
> > Sincerely,
> > Dmirtiy Pavlov
> >
> >
> >
> > чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>:
> >
> >> Guys,
> >>
> >> Thanks for responses. But I still fail to understand how it affects a
> user.
> >>
> >> Are you saying that with persistence enabled Random-LRU is always used
> >> regardless of what is specified in pageEvictionMode, while in in-memory
> >> only scenario we can also utilize Random-2-LRU for eviction? Is this
> >> accurate? Are there any other differences?
> >>
> >> -Val
> >>
> >> On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov 
> >> wrote:
> >>
> >>> Hi Vyacheslav,
> >>>
> >>> Yes, this is right, but for now PDS page based eviction uses
> Random-LRU,
> >>> not Random-2-LRU. Something may be changed in future Ignite releases,
> but
> >>> now Random-LRU is used. I am going to research if there is some room
> for
> >>> improvements in this aspect (e.g. IGNITE-7507 recenly found).
> >>>
> >>> If clean page is evicted, nothing really happends. If dirty page is
> >> evicted
> >>> from region during checkpointing process run, page is stored in file
> >> before
> >>> eviction
> >>>
> >>> Some info about this can be found in
> >>> https://cwiki.apache.org/confluence/display/IGNITE/
> >>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> >>> underthehood-Pagebasedeviction
> >>>
> >>> Sincerely,
> >>> Dmitriy Pavlov
> >>>
> >>> чт, 1 февр. 2018 г. в 9:53, Vyacheslav Daradur :

Re: Eviction policies with persistence

[Denis Magda]

Dmitriy,

I’m surprised to hear that Random-LRU works at the entry level when Ignite 
persistence is off. Frankly, this section confuses a lot:
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Entryeviction
 


While it was assumed that the entry-based eviction works only for on-heap 
caches:
https://apacheignite.readme.io/docs/evictions 


Alex G., please chime in and clarify the misunderstanding.

—
Denis

> On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov  wrote:
> 
> Hi Val,
> 
> I think it is quite accurate because eviction in case PDS enabled or not
> has quite different purposes.
> 
> 1. Let us consider PDS enabled and page eviction occurs. First of all it is
> page based eviction, but not entry-based. Actually data is not removed, but
> only written to disk. We can address this page later by ID.
> PDS eviction is primarily the replacement of pages from memory to page on
> disk. This eviction policy should ensure the maximum performance of the
> solution in the future. There is no data removal from grid in this case.
> And Ignite does not allow users to configure the policy. If benchmarks show
> that a change in policy results in increased performance, then we can
> switch policy.
> 
> 2. Entry-based eviction (if there is no persistence is available) is
> slithly different. User will need to reload data into grid in case entry is
> evicted but required in cache. In that case Ignite provides selection of
> policies.
> 
> Sincerely,
> Dmirtiy Pavlov
> 
> 
> 
> чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
> 
>> Guys,
>> 
>> Thanks for responses. But I still fail to understand how it affects a user.
>> 
>> Are you saying that with persistence enabled Random-LRU is always used
>> regardless of what is specified in pageEvictionMode, while in in-memory
>> only scenario we can also utilize Random-2-LRU for eviction? Is this
>> accurate? Are there any other differences?
>> 
>> -Val
>> 
>> On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov 
>> wrote:
>> 
>>> Hi Vyacheslav,
>>> 
>>> Yes, this is right, but for now PDS page based eviction uses Random-LRU,
>>> not Random-2-LRU. Something may be changed in future Ignite releases, but
>>> now Random-LRU is used. I am going to research if there is some room for
>>> improvements in this aspect (e.g. IGNITE-7507 recenly found).
>>> 
>>> If clean page is evicted, nothing really happends. If dirty page is
>> evicted
>>> from region during checkpointing process run, page is stored in file
>> before
>>> eviction
>>> 
>>> Some info about this can be found in
>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
>>> underthehood-Pagebasedeviction
>>> 
>>> Sincerely,
>>> Dmitriy Pavlov
>>> 
>>> чт, 1 февр. 2018 г. в 9:53, Vyacheslav Daradur :
>>> 
 Hi Valentin,
 
 As far as I know, when persistence is enabled and Ignite can't
 allocate new page-memory in the memory segment, then Ignite
 automatically evict some data from RAM to PDS using Random-2-LRU
 eviction algorithm. And there are no mechanisms to evict entries out
 when PDS is enabled.
 
 I'll be glad if I am not right and someone will correct me because, as
 Ignite user, I can't use PDS only as RAM disk replication for the use
 case in my project.
 
 
 On Thu, Feb 1, 2018 at 3:53 AM, Valentin Kulichenko
  wrote:
> Folks,
> 
> On "Eviction Policies" documentation page [1] we have the following
 callout:
> 
>> Configured eviction policy has no effect if Ignite persistence is
 enabled
>> Note that if Ignite Persistence is enabled, then the page-based
 evictions
> have no effect because the oldest pages will be purged from memory
> automatically.
> 
> This really confuses me. Why is there a difference in how data is
>>> evicted
> from memory depending on having disk enabled or not? And how does
 eviction
> work with persistence enabled then?
> 
> Does anyone can clarify?
> 
> [1] https://apacheignite.readme.io/docs/evictions
> 
> -Val
 
 
 
 --
 Best Regards, Vyacheslav D.
 
>>> 
>> 

Re: Eviction policies with persistence

[Lucas Beeler]

Hi Guys,

I agree with both Val and Dmitriy. Dmitriy is technically correct, but Val is 
also right that we should clean up the language.

Here’s what I mean by "cleaning up the language": unless I’m missing something, 
eviction policy has no effect when Ignite persistence is enabled, because 
Ignite will adaptively page-in and page-out memory to disk in a fashion very 
similar to how modern operating systems handle virtual memory. Using an 
explicitly configured eviction policy really only matters if you’re using 3rd 
party persistence (such as write-behind to an RDBMS). So basically, there are 
two different use cases:
Case #1: Native Persistence is On: eviction policy is meaningless because 
paging is handled automatically by Ignite. The actual algorithm used by Ignite 
doesn’t matter—so long as it works—because in this case Ignite is a magical 
black box that handles paging for you and you can’t configure it.
Case #2: Native Persistence is Off: in this case, users have to decide how to 
deal with out-of-memory situations by specifying their own eviction policy. 
Ignite will not magically page to disk and so users need to configure eviction. 
When configuring eviction, they have several configuration options.
So, in my mind, that documentation page needs to have a very clear sentence in 
its first paragraph that says something like “look, all of the stuff on this 
page only matters if you don’t have persistence enabled.”

Just my two cents…

Take care,
Lucas

--
Lucas BEELER
Technical Consultant, Professional Services
GridGain Systems
www.gridgain.com

> On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov  wrote:
> 
> Hi Val,
> 
> I think it is quite accurate because eviction in case PDS enabled or not
> has quite different purposes.
> 
> 1. Let us consider PDS enabled and page eviction occurs. First of all it is
> page based eviction, but not entry-based. Actually data is not removed, but
> only written to disk. We can address this page later by ID.
> PDS eviction is primarily the replacement of pages from memory to page on
> disk. This eviction policy should ensure the maximum performance of the
> solution in the future. There is no data removal from grid in this case.
> And Ignite does not allow users to configure the policy. If benchmarks show
> that a change in policy results in increased performance, then we can
> switch policy.
> 
> 2. Entry-based eviction (if there is no persistence is available) is
> slithly different. User will need to reload data into grid in case entry is
> evicted but required in cache. In that case Ignite provides selection of
> policies.
> 
> Sincerely,
> Dmirtiy Pavlov
> 
> 
> 
> чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
> 
>> Guys,
>> 
>> Thanks for responses. But I still fail to understand how it affects a user.
>> 
>> Are you saying that with persistence enabled Random-LRU is always used
>> regardless of what is specified in pageEvictionMode, while in in-memory
>> only scenario we can also utilize Random-2-LRU for eviction? Is this
>> accurate? Are there any other differences?
>> 
>> -Val
>> 
>> On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov 
>> wrote:
>> 
>>> Hi Vyacheslav,
>>> 
>>> Yes, this is right, but for now PDS page based eviction uses Random-LRU,
>>> not Random-2-LRU. Something may be changed in future Ignite releases, but
>>> now Random-LRU is used. I am going to research if there is some room for
>>> improvements in this aspect (e.g. IGNITE-7507 recenly found).
>>> 
>>> If clean page is evicted, nothing really happends. If dirty page is
>> evicted
>>> from region during checkpointing process run, page is stored in file
>> before
>>> eviction
>>> 
>>> Some info about this can be found in
>>> https://cwiki.apache.org/confluence/display/IGNITE/
>>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
>>> underthehood-Pagebasedeviction
>>> 
>>> Sincerely,
>>> Dmitriy Pavlov
>>> 
>>> чт, 1 февр. 2018 г. в 9:53, Vyacheslav Daradur :
>>> 
 Hi Valentin,
 
 As far as I know, when persistence is enabled and Ignite can't
 allocate new page-memory in the memory segment, then Ignite
 automatically evict some data from RAM to PDS using Random-2-LRU
 eviction algorithm. And there are no mechanisms to evict entries out
 when PDS is enabled.
 
 I'll be glad if I am not right and someone will correct me because, as
 Ignite user, I can't use PDS only as RAM disk replication for the use
 case in my project.
 
 
 On Thu, Feb 1, 2018 at 3:53 AM, Valentin Kulichenko
  wrote:
> Folks,
> 
> On "Eviction Policies" documentation page [1] we have the following
 callout:
> 
>> Configured eviction policy has no effect if Ignite persistence is
 enabled
>> Note that if Ignite Persistence is enabled, then the page-based
 evictions
> have no effect because 

Re: Eviction policies with persistence

[Dmitry Pavlov]

Hi Val,

I think it is quite accurate because eviction in case PDS enabled or not
has quite different purposes.

1. Let us consider PDS enabled and page eviction occurs. First of all it is
page based eviction, but not entry-based. Actually data is not removed, but
only written to disk. We can address this page later by ID.
PDS eviction is primarily the replacement of pages from memory to page on
disk. This eviction policy should ensure the maximum performance of the
solution in the future. There is no data removal from grid in this case.
And Ignite does not allow users to configure the policy. If benchmarks show
that a change in policy results in increased performance, then we can
switch policy.

2. Entry-based eviction (if there is no persistence is available) is
slithly different. User will need to reload data into grid in case entry is
evicted but required in cache. In that case Ignite provides selection of
policies.

Sincerely,
Dmirtiy Pavlov



чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> Guys,
>
> Thanks for responses. But I still fail to understand how it affects a user.
>
> Are you saying that with persistence enabled Random-LRU is always used
> regardless of what is specified in pageEvictionMode, while in in-memory
> only scenario we can also utilize Random-2-LRU for eviction? Is this
> accurate? Are there any other differences?
>
> -Val
>
> On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov 
> wrote:
>
> > Hi Vyacheslav,
> >
> > Yes, this is right, but for now PDS page based eviction uses Random-LRU,
> > not Random-2-LRU. Something may be changed in future Ignite releases, but
> > now Random-LRU is used. I am going to research if there is some room for
> > improvements in this aspect (e.g. IGNITE-7507 recenly found).
> >
> > If clean page is evicted, nothing really happends. If dirty page is
> evicted
> > from region during checkpointing process run, page is stored in file
> before
> > eviction
> >
> > Some info about this can be found in
> > https://cwiki.apache.org/confluence/display/IGNITE/
> > Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> > underthehood-Pagebasedeviction
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > чт, 1 февр. 2018 г. в 9:53, Vyacheslav Daradur :
> >
> > > Hi Valentin,
> > >
> > > As far as I know, when persistence is enabled and Ignite can't
> > > allocate new page-memory in the memory segment, then Ignite
> > > automatically evict some data from RAM to PDS using Random-2-LRU
> > > eviction algorithm. And there are no mechanisms to evict entries out
> > > when PDS is enabled.
> > >
> > > I'll be glad if I am not right and someone will correct me because, as
> > > Ignite user, I can't use PDS only as RAM disk replication for the use
> > > case in my project.
> > >
> > >
> > > On Thu, Feb 1, 2018 at 3:53 AM, Valentin Kulichenko
> > >  wrote:
> > > > Folks,
> > > >
> > > > On "Eviction Policies" documentation page [1] we have the following
> > > callout:
> > > >
> > > >> Configured eviction policy has no effect if Ignite persistence is
> > > enabled
> > > >> Note that if Ignite Persistence is enabled, then the page-based
> > > evictions
> > > > have no effect because the oldest pages will be purged from memory
> > > > automatically.
> > > >
> > > > This really confuses me. Why is there a difference in how data is
> > evicted
> > > > from memory depending on having disk enabled or not? And how does
> > > eviction
> > > > work with persistence enabled then?
> > > >
> > > > Does anyone can clarify?
> > > >
> > > > [1] https://apacheignite.readme.io/docs/evictions
> > > >
> > > > -Val
> > >
> > >
> > >
> > > --
> > > Best Regards, Vyacheslav D.
> > >
> >
>

Re: Eviction policies with persistence

[Valentin Kulichenko]

Guys,

Thanks for responses. But I still fail to understand how it affects a user.

Are you saying that with persistence enabled Random-LRU is always used
regardless of what is specified in pageEvictionMode, while in in-memory
only scenario we can also utilize Random-2-LRU for eviction? Is this
accurate? Are there any other differences?

-Val

On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov  wrote:

> Hi Vyacheslav,
>
> Yes, this is right, but for now PDS page based eviction uses Random-LRU,
> not Random-2-LRU. Something may be changed in future Ignite releases, but
> now Random-LRU is used. I am going to research if there is some room for
> improvements in this aspect (e.g. IGNITE-7507 recenly found).
>
> If clean page is evicted, nothing really happends. If dirty page is evicted
> from region during checkpointing process run, page is stored in file before
> eviction
>
> Some info about this can be found in
> https://cwiki.apache.org/confluence/display/IGNITE/
> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> underthehood-Pagebasedeviction
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 1 февр. 2018 г. в 9:53, Vyacheslav Daradur :
>
> > Hi Valentin,
> >
> > As far as I know, when persistence is enabled and Ignite can't
> > allocate new page-memory in the memory segment, then Ignite
> > automatically evict some data from RAM to PDS using Random-2-LRU
> > eviction algorithm. And there are no mechanisms to evict entries out
> > when PDS is enabled.
> >
> > I'll be glad if I am not right and someone will correct me because, as
> > Ignite user, I can't use PDS only as RAM disk replication for the use
> > case in my project.
> >
> >
> > On Thu, Feb 1, 2018 at 3:53 AM, Valentin Kulichenko
> >  wrote:
> > > Folks,
> > >
> > > On "Eviction Policies" documentation page [1] we have the following
> > callout:
> > >
> > >> Configured eviction policy has no effect if Ignite persistence is
> > enabled
> > >> Note that if Ignite Persistence is enabled, then the page-based
> > evictions
> > > have no effect because the oldest pages will be purged from memory
> > > automatically.
> > >
> > > This really confuses me. Why is there a difference in how data is
> evicted
> > > from memory depending on having disk enabled or not? And how does
> > eviction
> > > work with persistence enabled then?
> > >
> > > Does anyone can clarify?
> > >
> > > [1] https://apacheignite.readme.io/docs/evictions
> > >
> > > -Val
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
> >
>

Re: Eviction policies with persistence

[Dmitry Pavlov]

Hi Vyacheslav,

Yes, this is right, but for now PDS page based eviction uses Random-LRU,
not Random-2-LRU. Something may be changed in future Ignite releases, but
now Random-LRU is used. I am going to research if there is some room for
improvements in this aspect (e.g. IGNITE-7507 recenly found).

If clean page is evicted, nothing really happends. If dirty page is evicted
from region during checkpointing process run, page is stored in file before
eviction

Some info about this can be found in
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Pagebasedeviction

Sincerely,
Dmitriy Pavlov

чт, 1 февр. 2018 г. в 9:53, Vyacheslav Daradur :

> Hi Valentin,
>
> As far as I know, when persistence is enabled and Ignite can't
> allocate new page-memory in the memory segment, then Ignite
> automatically evict some data from RAM to PDS using Random-2-LRU
> eviction algorithm. And there are no mechanisms to evict entries out
> when PDS is enabled.
>
> I'll be glad if I am not right and someone will correct me because, as
> Ignite user, I can't use PDS only as RAM disk replication for the use
> case in my project.
>
>
> On Thu, Feb 1, 2018 at 3:53 AM, Valentin Kulichenko
>  wrote:
> > Folks,
> >
> > On "Eviction Policies" documentation page [1] we have the following
> callout:
> >
> >> Configured eviction policy has no effect if Ignite persistence is
> enabled
> >> Note that if Ignite Persistence is enabled, then the page-based
> evictions
> > have no effect because the oldest pages will be purged from memory
> > automatically.
> >
> > This really confuses me. Why is there a difference in how data is evicted
> > from memory depending on having disk enabled or not? And how does
> eviction
> > work with persistence enabled then?
> >
> > Does anyone can clarify?
> >
> > [1] https://apacheignite.readme.io/docs/evictions
> >
> > -Val
>
>
>
> --
> Best Regards, Vyacheslav D.
>

Re: Eviction policies with persistence

[Vyacheslav Daradur]

Hi Valentin,

As far as I know, when persistence is enabled and Ignite can't
allocate new page-memory in the memory segment, then Ignite
automatically evict some data from RAM to PDS using Random-2-LRU
eviction algorithm. And there are no mechanisms to evict entries out
when PDS is enabled.

I'll be glad if I am not right and someone will correct me because, as
Ignite user, I can't use PDS only as RAM disk replication for the use
case in my project.


On Thu, Feb 1, 2018 at 3:53 AM, Valentin Kulichenko
 wrote:
> Folks,
>
> On "Eviction Policies" documentation page [1] we have the following callout:
>
>> Configured eviction policy has no effect if Ignite persistence is enabled
>> Note that if Ignite Persistence is enabled, then the page-based evictions
> have no effect because the oldest pages will be purged from memory
> automatically.
>
> This really confuses me. Why is there a difference in how data is evicted
> from memory depending on having disk enabled or not? And how does eviction
> work with persistence enabled then?
>
> Does anyone can clarify?
>
> [1] https://apacheignite.readme.io/docs/evictions
>
> -Val



-- 
Best Regards, Vyacheslav D.

Eviction policies with persistence

[Valentin Kulichenko]

Folks,

On "Eviction Policies" documentation page [1] we have the following callout:

> Configured eviction policy has no effect if Ignite persistence is enabled
> Note that if Ignite Persistence is enabled, then the page-based evictions
have no effect because the oldest pages will be purged from memory
automatically.

This really confuses me. Why is there a difference in how data is evicted
from memory depending on having disk enabled or not? And how does eviction
work with persistence enabled then?

Does anyone can clarify?

[1] https://apacheignite.readme.io/docs/evictions

-Val