xuchuanyin resolved CARBONDATA-2238.
    Resolution: Fixed

> Optimization in unsafe sort during data loading
> -----------------------------------------------
>                 Key: CARBONDATA-2238
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-2238
>             Project: CarbonData
>          Issue Type: Improvement
>          Components: data-load
>            Reporter: xuchuanyin
>            Assignee: xuchuanyin
>            Priority: Major
>          Time Spent: 6h 10m
>  Remaining Estimate: 0h
> Inspired by batch_sort, if we have enough memory, in local_sort with unsafe 
> property, we can hold all the row pages in memory if possible and only spill 
> the pages to disk as sort temp file if the memory is unavailable.
> Before spilling the pages, we can do in-memory merge sort of the pages.
> Each time we request an unsafe row page, if the memory is unavailable, we can 
> trigger a merge sort for the in-memory pages and spill the result to disk as 
> a sort temp file. So the incoming pages will be held into the memory instead 
> of spilling to disk directly.
> After this implementation, the data size during each spilling will be bigger 
> than that of before and will benefit the disk IO.

This message was sent by Atlassian JIRA

Reply via email to