GitHub user jackylk opened a pull request:

    [CARBONDATA-318] Implement an InMemory Sorter that makes maximum usage of 
memory for data load

    Changed as following:
    1. Change to keep rows and sort in memory if memory is 
sufficient, otherwise spill to disk.
    2. Change SortKeyStep and MdkeyGenStep to support both in memory sort and 
merge sort.
    To choose between these two approaches, user can set SORT_SIZE in carbon 
property, like set it to 3 million rows:
      // Number of rows to keep in memory when loading data, if number of input 
row exceeds this value,
      // carbon will use merge sort instead of in memory sort

You can merge this pull request into a Git repository by running:

    $ git pull in-memory-sort

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #242
commit 23d9fbbca33cd5bb56a0a648596ad4d61c32fa04
Author: jackylk <>
Date:   2016-10-15T15:53:16Z

    add in memory sort in data load

commit 45dae7c4c7b535540bafb15bcc896061cfae7ca7
Author: jackylk <>
Date:   2016-10-15T17:55:09Z

    fix empty row


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

Reply via email to