[
https://issues.apache.org/jira/browse/CARBONDATA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacky Li resolved CARBONDATA-2901.
----------------------------------
Resolution: Fixed
Fix Version/s: 1.5.0
> Problem: Jvm crash in Load scenario when unsafe memory allocation is failed.
> ----------------------------------------------------------------------------
>
> Key: CARBONDATA-2901
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2901
> Project: CarbonData
> Issue Type: Bug
> Reporter: Ajantha Bhat
> Assignee: Ajantha Bhat
> Priority: Critical
> Fix For: 1.5.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Problem: Jvm crash in Load scenario when unsafe memory allocation is failed.
> scenario:
> a) Have many cores while loading. suggested more than 10.
> [carbon.number.of.cores.while.loading]
> b) Load huge data with local sort, more than 5GB (keeping default unsafe
> memory manager as 512 MB)
> c) when task failes due to not enough unsafae memory, JVM crashes with
> SIGSEGV.
> root casue:
> while sorting, all iterator threads are waiting at
> UnsafeSortDataRows.addRowBatch as all iterator works on one row page.
> Only one iterator thread will try to allocate memory. Before that it has
> freed current page in handlePreviousPage().
> When allocate memory failed, row page will still have that old reference.
> next thread will again use same reference and call handlePreviousPage().
> So, Jvm crashes as freed memory is accessed.
> solution:
> When allocation failed, set row page reference to null.
> So, that next thread will not do any operation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
