[
https://issues.apache.org/jira/browse/CARBONDATA-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kumar vishal resolved CARBONDATA-2889.
--------------------------------------
Resolution: Fixed
> Support Decoder based fall back mechanism in Local Dictionary
> -------------------------------------------------------------
>
> Key: CARBONDATA-2889
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2889
> Project: CarbonData
> Issue Type: Sub-task
> Reporter: Akash R Nilugal
> Assignee: Akash R Nilugal
> Priority: Major
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> Currently, when the fallback is initiated for a column page in case of local
> dictionary, we are keeping both encoded data
> and actual data in memory and then we form the new column page without
> dictionary encoding and then at last we free the Encoded Column Page.
> Because of this offheap memory footprint increases.
>
> We can reduce the offheap memory footprint. This can be done using decoder
> based fallback mechanism.
> This means, no need to keep the actual data along with encoded data in
> encoded column page. We can keep only encoded data and to form a new column
> page, get the dictionary data from encoded column page by uncompressing and
> using dictionary data get the actual data using local dictionary generator
> and put it in new column page created and compress it again and give to
> consumer for writing blocklet.
>
> The above process may slow down the loading, but it will reduces the memory
> footprint. So we can give a property which will decide whether to take
> current fallback procedure or decoder based fallback mechanism dring fallback
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
