ASF GitHub Bot commented on CARBONDATA-318:

GitHub user jackylk opened a pull request:


    [CARBONDATA-318] Implement an InMemory Sorter that makes maximum usage of 
memory for data load

    Changed as following:
    1. Change SortDataRows.java to keep rows and sort in memory if memory is 
sufficient, otherwise spill to disk.
    2. Change SortKeyStep and MdkeyGenStep to support both in memory sort and 
merge sort.
    To choose between these two approaches, user can set SORT_SIZE in carbon 
property, like set it to 3 million rows:
      // Number of rows to keep in memory when loading data, if number of input 
row exceeds this value,
      // carbon will use merge sort instead of in memory sort

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jackylk/incubator-carbondata in-memory-sort

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #242
commit 23d9fbbca33cd5bb56a0a648596ad4d61c32fa04
Author: jackylk <jacky.li...@huawei.com>
Date:   2016-10-15T15:53:16Z

    add in memory sort in data load

commit 45dae7c4c7b535540bafb15bcc896061cfae7ca7
Author: jackylk <jacky.li...@huawei.com>
Date:   2016-10-15T17:55:09Z

    fix empty row


> Implement an InMemory Sorter that makes maximum usage of memory while sorting
> -----------------------------------------------------------------------------
>                 Key: CARBONDATA-318
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-318
>             Project: CarbonData
>          Issue Type: Sub-task
>            Reporter: Jacky Li
>            Assignee: Jacky Li
>             Fix For: 0.2.0-incubating
> Change SortDataRows into an External Sorter, it should sort in memory until 
> it reach configured size, then spill to disk. It should provide following 
> interface:
> 1. addRow:  insert rows into the sorter.
> 2. getIterator: will return an iterator that iterates on sorted rows, the 
> sorted row could come from memory or files

This message was sent by Atlassian JIRA

Reply via email to