[
https://issues.apache.org/jira/browse/CARBONDATA-484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15760527#comment-15760527
]
Ben Manes commented on CARBONDATA-484:
--------------------------------------
You might consider using TinyLFU instead of LRU to account for frequency. See
this [HighScalability
article](http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html)
that provides a short description. In short, a frequency filter is placed in
front of the LRU policy to avoid low value entries from polluting the cache.
> Implement LRU cache for B-Tree
> -------------------------------
>
> Key: CARBONDATA-484
> URL: https://issues.apache.org/jira/browse/CARBONDATA-484
> Project: CarbonData
> Issue Type: New Feature
> Reporter: Mohammad Shahid Khan
> Assignee: Mohammad Shahid Khan
> Attachments: B-Tree LRU Cache.pdf
>
>
> LRU Cache for B-Tree is proposed to ensure to avoid out memory, when too
> many number of tables exits and all are not frequently used.
> Problem:
> CarbonData is maintaining two level of B-Tree cache, one at the driver level
> and another at executor level. Currently CarbonData has the mechanism to
> invalidate the segments and blocks cache for the invalid table segments, but
> there is no eviction policy for the unused cached object. So the instance at
> which complete memory is utilized then the system will not be able to process
> any new requests.
> Solution:
> In the cache maintained at the driver level and at the executor there must be
> objects in cache currently not in use. Therefore system should have the
> mechanism to below mechanism.
> 1. Set the max memory limit till which objects could be hold in the
> memory.
> 2. When configured memory limit reached then identify the cached
> objects currently not in use so that the required memory could be freed
> without impacting the existing process.
> 3. Eviction should be done only till the required memory is not meet.
> For details please refer to attachments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
