[
https://issues.apache.org/jira/browse/TAJO-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084267#comment-14084267
]
Hudson commented on TAJO-987:
-----------------------------
SUCCESS: Integrated in Tajo-master-build #322 (See
[https://builds.apache.org/job/Tajo-master-build/322/])
TAJO-987: Hash shuffle should be balanced according to intermediate volumes.
(hyunsik: rev eeaf379a48030dd819a6daf2040c779379543ac8)
* tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
* CHANGES
* tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java
> Hash shuffle should be balanced according to intermediate volumes
> -----------------------------------------------------------------
>
> Key: TAJO-987
> URL: https://issues.apache.org/jira/browse/TAJO-987
> Project: Tajo
> Issue Type: Improvement
> Components: data shuffle
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.9.0
>
>
> It is not hard to see skewed data set in practice. Currently, hash shuffled
> intermediate are performed by distributing partition keys without considering
> their partition volumes. As a result, with skewed intermediate data, a few of
> nodes are likely to take much longer time than most of all nodes. It can
> cause performance degradation. We need some solution to mitigate this problem.
> This patch assigns the intermediate data by balancing their volumes. The
> approach is a kind of greedy algorithm. In many cases, the shuffle num can be
> over tens of thousands. I also considered the computation complexity. Its
> complexity is O \(n\). It will show reasonable performance and balanced
> results.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
