[
https://issues.apache.org/jira/browse/CARBONDATA-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xuchuanyin resolved CARBONDATA-2246.
------------------------------------
Resolution: Fixed
> Fix not-enough-memory bugs in unsafe data loading
> -------------------------------------------------
>
> Key: CARBONDATA-2246
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2246
> Project: CarbonData
> Issue Type: Bug
> Reporter: xuchuanyin
> Assignee: xuchuanyin
> Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Currently in carbon data loading, if we enable the unsafe loading and specify
> corresponding properties, data loading will end in OOM.
> The key properties to reproduce the bug are as following:
> ```
> 01:
> CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_INMEMORY_MERGE_SORT,
> "true")
> CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_UNSAFE_SORT,
> "true")
> 02:
> 03: // unsafe sort memory manager
> 04:
> CarbonProperties.getInstance().addProperty(CarbonCommonConstants.IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB,
> "1024")
> 05:
> 06: // unsafe working memory manager
> 07:
> CarbonProperties.getInstance().addProperty(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB,
> "512")
> 08:
> 09: // one unsafe page, better if loading_cores * this < memory
> 10:
> CarbonProperties.getInstance().addProperty(CarbonCommonConstants.OFFHEAP_SORT_CHUNK_SIZE_IN_MB,
> "512")
> ```
>
> Notice that the `OFFHEAP_SORT_CHUNK_SIZE_IN_MB` are exactly the same as
> `UNSAFE_WORKING_MEMORY_IN_MB` which will cause problem
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
