Stephen It seems the problem is how ignite deals with this was not clear to me. To make sure I understand this correctly: Ignite with native persistence on, automatically overflows pages to disk using the least recently used policy - LRU. (which is great) Pages could be a partial row in a table. But that's ok because a SQL query that includes that will pull from RAM and disk automatically under the covers. i.e. it's all auto-magically done for us.
What is still not clear is, as branches are deleted or we scale-up servers and more memory is then made available, will it rotate back into memory from disk what was rotated out? assuming reverse LRU order? Thanks!! Ryan On Tue, Jan 19, 2021 at 7:28 AM Stephen Darlington < [email protected]> wrote: > As long as new branches — to use your analogy — are in memory, why does it > matter that a few others are too? The least recently used branches will > automatically (LRU) be purged from memory if space is needed for new > branches. > > In fact, if you’re worried about available memory, a *time based* eviction > policy won’t work. What if you expect to have 100 branches and size your > cluster appropriately. Suddenly, 1000 branches are created. Boom! > > With a space-based eviction policy — as is the default with Ignite native > persistence — that works just fine. > > You could create a cache with an eviction policy. When the records are > deleted after a week, you can have a process listening to delete events and > copy the record to a different cache in a small data region. > > So what you’re asking for is possible, it’s just more complicated and less > effective than the alternative. > > On 19 Jan 2021, at 14:04, Ryan Trollip <[email protected]> wrote: > > Stephen > > Let's use an analogy of projects in source control. Let's say we have a > very active community of developers, they are creating 100 new branches a > day. Each branch has a few thousand objects and associated properties etc. > but these developers don't clean up by deleting branches. > We want new branches to be cached in memory and available high performance > read and write, but older branches to go to disk to save on memory hardware > needs, since many are abandoned. > The policy could read something like this: Branches that have not been > accessed in 1 week, move to disk. On branch access, if on disk, move back > to RAM. > > Thanks > Ryan > > On Tue, Jan 19, 2021 at 2:33 AM Stephen Darlington < > [email protected]> wrote: > >> I guess I’m still not clear why you need to explicitly remove them from >> memory. >> >> By virtue of using native persistence, they’re already on disk. If you >> load new data, the old entries will eventually be flushed from memory (but >> remain on disk). What do you gain by removing entries from memory at a >> specific time? >> >> Regards, >> Stephen >> >> > On 19 Jan 2021, at 06:02, Naveen <[email protected]> wrote: >> > >> > Hi Stephen >> > >> > on the same mail chain, we also data like OTP (one time passwds) which >> are >> > no relevant after a while, but we dont want to expire or delete them, >> just >> > get them flushed to disk, like wise we do have other requirements where >> data >> > is very relevant only for a certain duration, later on its not >> important. >> > Thats the whole idea of exploring eviction policies >> > >> > Naveen >> > >> > >> > >> > -- >> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >> >> > >
