GitHub user xuchuanyin reopened a pull request:

    https://github.com/apache/carbondata/pull/1808

    [CARBONDATA-2023][DataLoad] Add size base block allocation in data loading

    Carbondata assign blocks to nodes at the beginning of data loading.
    Previous block allocation strategy is block number based and it will
    suffer skewed data problem if the size of input files differs a lot.
    
    We introduced a size based block allocation strategy to optimize data
    loading performance in skewed data scenario.
    
    Be sure to do all of the following checklist to help us incorporate 
    your contribution quickly and easily:
    
     - [x] Any interfaces changed?
     `Only changed the internal interfaces`
     - [x] Any backward compatibility impacted?
     `No`
     - [x] Document update required?
    `Updated the document`
     - [x] Testing done
            Please provide details on 
            - Whether new unit test cases have been added or why no new tests 
are required?
    `Added tests to verify the block-allocation correctness`
            - How it is tested? Please attach test report.
    `Tested in local 3-node cluster`
            - Is it a performance related change? Please attach the performance 
test report.
    ```
    In my scenario, the size of input data file varies from 1KB to about 5GB. 
    Before enabling this feature, each executor processed the same number of 
blocks
     and the processed data size had a 5X gap.  --(block number based 
allocation)
    After enabling this feature, each executor processed almost the same size 
of data
     and the processed number of data blocks had 6X gap. -- (block size based 
allocation)
    
    The data loading performance had been promoted from 41MB/s/Node to 
61MB/s/Node, 
    about 50% performance enhancement gained.
    ```
    
            - Any additional information to help reviewers in testing this 
change.
           `I refactored the code to make it more readable. The core code 
mainly lies in CarbonLoaderUtil`
     - [x] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
    `Not related`


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xuchuanyin/carbondata 
opt_size_base_block_allocation

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1808.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1808
    
----
commit eec35eae52affdfdbd05915ece2f19bedec5e310
Author: xuchuanyin <xuchuanyin@...>
Date:   2018-02-08T06:42:39Z

    Add size based block allocation strategy in data loading
    
    Carbondata assign blocks to nodes at the beginning of data loading.
    Previous block allocation strategy is block number based and it will
    shuffer skewed data problem if the size of input files differs a lot.
    
    We introduced a size based block allocation strategy to optimize data
    loading performance in skewed data scenario.

commit da4f93dd7bbe9faa045c751cba7ae1dd22ce12e4
Author: xuchuanyin <xuchuanyin@...>
Date:   2018-02-09T02:25:42Z

    Fix review comments

----


---

Reply via email to