Ajantha Bhat created CARBONDATA-2901:
----------------------------------------
Summary: Problem: Jvm crash in Load scenario when unsafe memory
allocation is failed.
Key: CARBONDATA-2901
URL: https://issues.apache.org/jira/browse/CARBONDATA-2901
Project: CarbonData
Issue Type: Bug
Reporter: Ajantha Bhat
Assignee: Ajantha Bhat
Problem: Jvm crash in Load scenario when unsafe memory allocation is failed.
scenario:
a) Have many cores while loading. suggested more than 10.
[carbon.number.of.cores.while.loading]
b) Load huge data with local sort, more than 5GB (keeping default unsafe memory
manager as 512 MB)
c) when task failes due to not enough unsafae memory, JVM crashes with SIGSEGV.
root casue:
while sorting, all iterator threads are waiting at
UnsafeSortDataRows.addRowBatch as all iterator works on one row page.
Only one iterator thread will try to allocate memory. Before that it has freed
current page in handlePreviousPage().
When allocate memory failed, row page will still have that old reference. next
thread will again use same reference and call handlePreviousPage().
So, Jvm crashes as freed memory is accessed.
solution:
When allocation failed, set row page reference to null.
So, that next thread will not do any operation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
