[
https://issues.apache.org/jira/browse/TAJO-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084226#comment-14084226
]
ASF GitHub Bot commented on TAJO-987:
-------------------------------------
Github user jihoonson commented on a diff in the pull request:
https://github.com/apache/tajo/pull/101#discussion_r15738646
--- Diff:
tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
---
@@ -18,7 +18,9 @@
package org.apache.tajo.master.querymaster;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
--- End diff --
This is an unused imports.
Please remove it.
> Hash shuffle should be balanced according to intermediate volumes
> -----------------------------------------------------------------
>
> Key: TAJO-987
> URL: https://issues.apache.org/jira/browse/TAJO-987
> Project: Tajo
> Issue Type: Bug
> Components: data shuffle
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> Fix For: 0.9.0
>
>
> It is not hard to see skewed data set in practice. Currently, hash shuffled
> intermediate are performed by distributing partition keys without considering
> their partition volumes. As a result, with skewed intermediate data, a few of
> nodes are likely to take much longer time than most of all nodes. It can
> cause performance degradation. We need some solution to mitigate this problem.
> This patch assigns the intermediate data by balancing their volumes. The
> approach is a kind of greedy algorithm. In many cases, the shuffle num can be
> over tens of thousands. I also considered the computation complexity. Its
> complexity is O \(n\). It will show reasonable performance and balanced
> results.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
