[jira] [Updated] (CARBONDATA-484) LRU cache for B-Tree to ensure to avoid out memory, when too many number of tables exits and all are not frequently used.
[ https://issues.apache.org/jira/browse/CARBONDATA-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-484: Attachment: B-Tree LRU Cache.pdf LRU cache for B-Tree Design Doc > LRU cache for B-Tree to ensure to avoid out memory, when too many number of > tables exits and all are not frequently used. > - > > Key: CARBONDATA-484 > URL: https://issues.apache.org/jira/browse/CARBONDATA-484 > Project: CarbonData > Issue Type: New Feature >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > Attachments: B-Tree LRU Cache.pdf > > > *LRU Cache for B-Tree* > Problem: > CarbonData is maintaining two level of B-Tree cache, one at the driver level > and another at executor level. Currently CarbonData has the mechanism to > invalidate the segments and blocks cache for the invalid table segments, but > there is no eviction policy for the unused cached object. So the instance at > which complete memory is utilized then the system will not be able to process > any new requests. > Solution: > In the cache maintained at the driver level and at the executor there must be > objects in cache currently not in use. Therefore system should have the > mechanism to below mechanism. > 1. Set the max memory limit till which objects could be hold in the > memory. > 2. When configured memory limit reached then identify the cached > objects currently not in use so that the required memory could be freed > without impacting the existing process. > 3. Eviction should be done only till the required memory is not meet. > For details please refer to attachments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-484) LRU cache for B-Tree to ensure to avoid out memory, when too many number of tables exits and all are not frequently used.
[ https://issues.apache.org/jira/browse/CARBONDATA-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Shahid Khan updated CARBONDATA-484: Issue Type: New Feature (was: Bug) > LRU cache for B-Tree to ensure to avoid out memory, when too many number of > tables exits and all are not frequently used. > - > > Key: CARBONDATA-484 > URL: https://issues.apache.org/jira/browse/CARBONDATA-484 > Project: CarbonData > Issue Type: New Feature >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan > > *LRU Cache for B-Tree* > Problem: > CarbonData is maintaining two level of B-Tree cache, one at the driver level > and another at executor level. Currently CarbonData has the mechanism to > invalidate the segments and blocks cache for the invalid table segments, but > there is no eviction policy for the unused cached object. So the instance at > which complete memory is utilized then the system will not be able to process > any new requests. > Solution: > In the cache maintained at the driver level and at the executor there must be > objects in cache currently not in use. Therefore system should have the > mechanism to below mechanism. > 1. Set the max memory limit till which objects could be hold in the > memory. > 2. When configured memory limit reached then identify the cached > objects currently not in use so that the required memory could be freed > without impacting the existing process. > 3. Eviction should be done only till the required memory is not meet. > For details please refer to attachments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
