Manish Gupta created CARBONDATA-2638:
----------------------------------------
Summary: Implement driver min max caching for specified columns
and segregate block and blocklet cache
Key: CARBONDATA-2638
URL: https://issues.apache.org/jira/browse/CARBONDATA-2638
Project: CarbonData
Issue Type: New Feature
Reporter: Manish Gupta
Assignee: Manish Gupta
Attachments: Driver_Block_Cache.docx
*Background*
Current implementation of Blocklet dataMap caching in driver is that it caches
the min and max values of all the columns in schema by default.
*Problem*
Problem with this implementation is that as the number of loads increases the
memory required to hold min and max values also increases considerably. We know
that in most of the scenarios there is a single driver and memory configured
for driver is less as compared to executor. With continuous increase in memory
requirement driver can even go out of memory which makes the situation further
worse.
*Solution*
1. Cache only the required columns in Driver
2. Segregation of block and Blocklet level cache**
For more details please check the attached document
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
