GitHub user ala opened a pull request:

    [SPARK-23496][CORE] Locality of coalesced partitions can be severely skewed 
by the order of input partitions

    ## What changes were proposed in this pull request?
    The algorithm in `DefaultPartitionCoalescer.setupGroups` is responsible for 
picking preferred locations for coalesced partitions. It analyzes the preferred 
locations of input partitions. It starts by trying to create one partition for 
each unique location in the input. However, if the the requested number of 
coalesced partitions is higher that the number of unique locations, it has to 
pick duplicate locations.
    Previously, the duplicate locations would be picked by iterating over the 
input partitions in order, and copying their preferred locations to coalesced 
partitions. If the input partitions were clustered by location, this could 
result in severe skew.
    With the fix, instead of iterating over the list of input partitions in 
order, we pick them at random. It's not perfectly balanced, but it's much 
    ## How was this patch tested?
    Unit test reproducing the behavior was added.

You can merge this pull request into a Git repository by running:

    $ git pull SPARK-23496

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20664
commit 6d67dfc1d4c012492b97873beaac5a7cbfd6f55a
Author: Ala Luszczak <ala@...>
Date:   2018-02-23T14:37:19Z

    Fix SPARK-23496.



To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to