[
https://issues.apache.org/jira/browse/CARBONDATA-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Shahid Khan updated CARBONDATA-484:
--------------------------------------------
Summary: Implement LRU cache for B-Tree to ensure to avoid out memory, when
too many number of tables exits and all are not frequently used. (was: LRU
cache for B-Tree to ensure to avoid out memory, when too many number of tables
exits and all are not frequently used.)
> Implement LRU cache for B-Tree to ensure to avoid out memory, when too many
> number of tables exits and all are not frequently used.
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: CARBONDATA-484
> URL: https://issues.apache.org/jira/browse/CARBONDATA-484
> Project: CarbonData
> Issue Type: New Feature
> Reporter: Mohammad Shahid Khan
> Assignee: Mohammad Shahid Khan
> Attachments: B-Tree LRU Cache.pdf
>
>
> *LRU Cache for B-Tree*
> Problem:
> CarbonData is maintaining two level of B-Tree cache, one at the driver level
> and another at executor level. Currently CarbonData has the mechanism to
> invalidate the segments and blocks cache for the invalid table segments, but
> there is no eviction policy for the unused cached object. So the instance at
> which complete memory is utilized then the system will not be able to process
> any new requests.
> Solution:
> In the cache maintained at the driver level and at the executor there must be
> objects in cache currently not in use. Therefore system should have the
> mechanism to below mechanism.
> 1. Set the max memory limit till which objects could be hold in the
> memory.
> 2. When configured memory limit reached then identify the cached
> objects currently not in use so that the required memory could be freed
> without impacting the existing process.
> 3. Eviction should be done only till the required memory is not meet.
> For details please refer to attachments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
