GitHub user xuchuanyin opened a pull request:

    https://github.com/apache/carbondata/pull/2052

    [CARBONDATA-2246][DataLoad] Fix exhausted memory problem during unsafe data 
loading

    If the size of unsafe row page equals to that of working memory, the
    last page will exhaust the working memory and carbondata will suffer 'not
    enough memory' problem when we convert data to columnar format.
    
    All unsafe pages should be spilled to disk or moved to unsafe sort memory
    instead of be kept in unsafe working memory.
    
    Be sure to do all of the following checklist to help us incorporate 
    your contribution quickly and easily:
    
     - [x] Any interfaces changed?
     `NO`
     - [x] Any backward compatibility impacted?
      `NO`
     - [x] Document update required?
     `NO`
     - [x] Testing done
            Please provide details on 
            - Whether new unit test cases have been added or why no new tests 
are required?
    `Tests added`
            - How it is tested? Please attach test report.
    `Tested in local machine`
            - Is it a performance related change? Please attach the performance 
test report.
    `NO`
            - Any additional information to help reviewers in testing this 
change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xuchuanyin/carbondata 0312_bug_unsafe_memory

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2052.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2052
    
----
commit da3329ca60250e07b568f905ba5cdaa115f8d522
Author: xuchuanyin <xuchuanyin@...>
Date:   2018-03-12T12:25:17Z

    Fix exhausted memory problem during unsafe data loading
    
    If the size of unsafe row page equals to that of working memory, the
    last page will exhaust the working memory and carbondata will suffer 'not
    enough memory' problem when we convert data to columnar format.
    
    All unsafe pages should be spilled to disk or moved to unsafe sort memory
    instead of be kept in unsafe working memory.

----


---

Reply via email to