Guys, thanks for all the clarification, it becomes more clear now. I second Denis' question though.
And I actually don't think that we need to rename anything here. The concept of eviction as "freeing up memory when a user hits the memory limit" makes sense to me with both persistence enabled and disabled, I would not overcomplicate it. What we need to do os properly describe it. The confusion is caused only by the way documentation is written, in particular these two points: 1. We claim that "Configured eviction policy has no effect if Ignite persistence is enabled" which to me sounds like there is no eviction in this case at all. So it's completely unclear how memory is freed up when persistence is used and what kind of control user has. 2. We claim that we have "page-based eviction for the off-heap memory data sets" and "entry-based eviction for the optional on-heap cache" which based on this discussion just doesn't seem to be true. I think we need to have three separate sections here: one for off-heap with persistence, another for off-heap without persistence, and the last one for on-heap cache. Each section should provide detailed explanation of how memory is cleaned up in this particular case and what options user has to configure. In addition, it makes sense to have warnings in case configuration is used incorrectly. E.g. if pageEvictionMode is configured with persistence, or if evictionPolicy is configured without on-heap cache. -Val On Tue, Feb 6, 2018 at 1:23 PM, Denis Magda <dma...@apache.org> wrote: > Alexey, > > Let me understand this part because it’s still not crystal clear to me: > > > Now as per the > > eviction mechanism, it is both per-page and per-entry: first, we choose a > > page which fits most for eviction, however, we cannot simply erase the > page > > because quite a lot of data structures are referring to that page. To > deal > > with it, we read keys that the chosen page contains and then clear > > individual cache entries, which in turn will clear up all necessary > links. > > > However will all the keys-values from a chosen page be removed eventually > during the eviction phase? If it’s true then it’s still a page based > eviction - we select 5 random oldest pages and purge all the key-values > from them. > > — > Denis > > > On Feb 4, 2018, at 12:05 PM, Alexey Goncharuk < > alexey.goncha...@gmail.com> wrote: > > > > Guys, > > > > To clarify the questions, I would like to reiterate over the mechanics of > > evictions and then suggest a renaming that should resolve such things in > > the future. > > > > First of all, Lucas is right - eviction policy only makes sense when > native > > persistence is disabled because what it does is actually freeing up > memory > > when a user hits the memory limit. The only way to do this is to destroy > > inserted data because there is no other way to free memory. Now as per > the > > eviction mechanism, it is both per-page and per-entry: first, we choose a > > page which fits most for eviction, however, we cannot simply erase the > page > > because quite a lot of data structures are referring to that page. To > deal > > with it, we read keys that the chosen page contains and then clear > > individual cache entries, which in turn will clear up all necessary > links. > > If there are no concurrent updates, the page becomes empty and will be > > reused for other user data. This is exactly what is explained on the wiki > > page (at least in my reading of the page). > > > > Second, at this point, I would rename page management mechanism with > > enabled persistence from 'eviction' to 'page replacement' or 'page > > swapping', because it does not destroy any data, but rather replaces a > > chosen buffer in memory from one page to another. The content of neither > > pages does not change during page replacement. This mechanism is unlikely > > to be selected by a user because the effectiveness of page replacements > > heavily depends on internal data structures and may change from version > to > > version, and may even be adaptive depending on the load pattern. > > > > Hope this resolves the confusion. > > > > --AG > > > > 2018-02-03 1:03 GMT+03:00 Denis Magda <dma...@apache.org>: > > > >> Dmitriy, > >> > >> I’m surprised to hear that Random-LRU works at the entry level when > Ignite > >> persistence is off. Frankly, this section confuses a lot: > >> https://cwiki.apache.org/confluence/display/IGNITE/ > >> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory- > >> underthehood-Entryeviction <https://cwiki.apache.org/ > >> confluence/display/IGNITE/Ignite+Durable+Memory+-+under+ > >> the+hood#IgniteDurableMemory-underthehood-Entryeviction> > >> > >> While it was assumed that the entry-based eviction works only for > on-heap > >> caches: > >> https://apacheignite.readme.io/docs/evictions < > >> https://apacheignite.readme.io/docs/evictions> > >> > >> Alex G., please chime in and clarify the misunderstanding. > >> > >> — > >> Denis > >> > >>> On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov <dpavlov....@gmail.com> > wrote: > >>> > >>> Hi Val, > >>> > >>> I think it is quite accurate because eviction in case PDS enabled or > not > >>> has quite different purposes. > >>> > >>> 1. Let us consider PDS enabled and page eviction occurs. First of all > it > >> is > >>> page based eviction, but not entry-based. Actually data is not removed, > >> but > >>> only written to disk. We can address this page later by ID. > >>> PDS eviction is primarily the replacement of pages from memory to page > on > >>> disk. This eviction policy should ensure the maximum performance of the > >>> solution in the future. There is no data removal from grid in this > case. > >>> And Ignite does not allow users to configure the policy. If benchmarks > >> show > >>> that a change in policy results in increased performance, then we can > >>> switch policy. > >>> > >>> 2. Entry-based eviction (if there is no persistence is available) is > >>> slithly different. User will need to reload data into grid in case > entry > >> is > >>> evicted but required in cache. In that case Ignite provides selection > of > >>> policies. > >>> > >>> Sincerely, > >>> Dmirtiy Pavlov > >>> > >>> > >>> > >>> чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko < > >>> valentin.kuliche...@gmail.com>: > >>> > >>>> Guys, > >>>> > >>>> Thanks for responses. But I still fail to understand how it affects a > >> user. > >>>> > >>>> Are you saying that with persistence enabled Random-LRU is always used > >>>> regardless of what is specified in pageEvictionMode, while in > in-memory > >>>> only scenario we can also utilize Random-2-LRU for eviction? Is this > >>>> accurate? Are there any other differences? > >>>> > >>>> -Val > >>>> > >>>> On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov <dpavlov....@gmail.com> > >>>> wrote: > >>>> > >>>>> Hi Vyacheslav, > >>>>> > >>>>> Yes, this is right, but for now PDS page based eviction uses > >> Random-LRU, > >>>>> not Random-2-LRU. Something may be changed in future Ignite releases, > >> but > >>>>> now Random-LRU is used. I am going to research if there is some room > >> for > >>>>> improvements in this aspect (e.g. IGNITE-7507 recenly found). > >>>>> > >>>>> If clean page is evicted, nothing really happends. If dirty page is > >>>> evicted > >>>>> from region during checkpointing process run, page is stored in file > >>>> before > >>>>> eviction > >>>>> > >>>>> Some info about this can be found in > >>>>> https://cwiki.apache.org/confluence/display/IGNITE/ > >>>>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory- > >>>>> underthehood-Pagebasedeviction > >>>>> > >>>>> Sincerely, > >>>>> Dmitriy Pavlov > >>>>> > >>>>> чт, 1 февр. 2018 г. в 9:53, Vyacheslav Daradur <daradu...@gmail.com > >: > >>>>> > >>>>>> Hi Valentin, > >>>>>> > >>>>>> As far as I know, when persistence is enabled and Ignite can't > >>>>>> allocate new page-memory in the memory segment, then Ignite > >>>>>> automatically evict some data from RAM to PDS using Random-2-LRU > >>>>>> eviction algorithm. And there are no mechanisms to evict entries out > >>>>>> when PDS is enabled. > >>>>>> > >>>>>> I'll be glad if I am not right and someone will correct me because, > as > >>>>>> Ignite user, I can't use PDS only as RAM disk replication for the > use > >>>>>> case in my project. > >>>>>> > >>>>>> > >>>>>> On Thu, Feb 1, 2018 at 3:53 AM, Valentin Kulichenko > >>>>>> <valentin.kuliche...@gmail.com> wrote: > >>>>>>> Folks, > >>>>>>> > >>>>>>> On "Eviction Policies" documentation page [1] we have the following > >>>>>> callout: > >>>>>>> > >>>>>>>> Configured eviction policy has no effect if Ignite persistence is > >>>>>> enabled > >>>>>>>> Note that if Ignite Persistence is enabled, then the page-based > >>>>>> evictions > >>>>>>> have no effect because the oldest pages will be purged from memory > >>>>>>> automatically. > >>>>>>> > >>>>>>> This really confuses me. Why is there a difference in how data is > >>>>> evicted > >>>>>>> from memory depending on having disk enabled or not? And how does > >>>>>> eviction > >>>>>>> work with persistence enabled then? > >>>>>>> > >>>>>>> Does anyone can clarify? > >>>>>>> > >>>>>>> [1] https://apacheignite.readme.io/docs/evictions > >>>>>>> > >>>>>>> -Val > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Best Regards, Vyacheslav D. > >>>>>> > >>>>> > >>>> > >> > >> > >
