[jira] [Commented] (HELIX-635) GenericTaskAssignmentCalculator does not balance task assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514749#comment-15514749
 ] 

ASF GitHub Bot commented on HELIX-635:
--

Github user asfgit closed the pull request at:

https://github.com/apache/helix/pull/51


> GenericTaskAssignmentCalculator does not balance task assignment
> 
>
> Key: HELIX-635
> URL: https://issues.apache.org/jira/browse/HELIX-635
> Project: Apache Helix
>  Issue Type: Bug
>  Components: helix-core
>Reporter: Junkai Xue
>Assignee: Junkai Xue
> Fix For: 0.6.x
>
>
> Currently, GenericTaskAssignmentCalculator using AutoRebalancerStrategy 
> compute the assignment mapping. There is a bug that all the tasks from same 
> job will be assigned to one node, which is not balanced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-635) GenericTaskAssignmentCalculator does not balance task assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504888#comment-15504888
 ] 

ASF GitHub Bot commented on HELIX-635:
--

Github user dasahcc commented on a diff in the pull request:

https://github.com/apache/helix/pull/51#discussion_r79503640
  
--- Diff: helix-core/src/main/java/org/apache/helix/task/JobConfig.java ---
@@ -449,11 +446,6 @@ public Builder setMaxAttemptsPerTask(int v) {
   return this;
 }
 
-public Builder setMaxForcedReassignmentsPerTask(int v) {
--- End diff --

Fixed


> GenericTaskAssignmentCalculator does not balance task assignment
> 
>
> Key: HELIX-635
> URL: https://issues.apache.org/jira/browse/HELIX-635
> Project: Apache Helix
>  Issue Type: Bug
>  Components: helix-core
>Reporter: Junkai Xue
>Assignee: Junkai Xue
> Fix For: 0.6.x
>
>
> Currently, GenericTaskAssignmentCalculator using AutoRebalancerStrategy 
> compute the assignment mapping. There is a bug that all the tasks from same 
> job will be assigned to one node, which is not balanced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-635) GenericTaskAssignmentCalculator does not balance task assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504890#comment-15504890
 ] 

ASF GitHub Bot commented on HELIX-635:
--

Github user dasahcc commented on a diff in the pull request:

https://github.com/apache/helix/pull/51#discussion_r79503677
  
--- Diff: helix-core/src/main/java/org/apache/helix/task/JobConfig.java ---
@@ -272,8 +261,6 @@ public TaskConfig getTaskConfig(String id) {
 }
 cfgMap.put(JobConfigProperty.TimeoutPerPartition.name(), "" + 
_timeoutPerTask);
 cfgMap.put(JobConfigProperty.MaxAttemptsPerTask.name(), "" + 
_maxAttemptsPerTask);
-cfgMap.put(JobConfigProperty.MaxForcedReassignmentsPerTask.name(),
--- End diff --

Fixed


> GenericTaskAssignmentCalculator does not balance task assignment
> 
>
> Key: HELIX-635
> URL: https://issues.apache.org/jira/browse/HELIX-635
> Project: Apache Helix
>  Issue Type: Bug
>  Components: helix-core
>Reporter: Junkai Xue
>Assignee: Junkai Xue
> Fix For: 0.6.x
>
>
> Currently, GenericTaskAssignmentCalculator using AutoRebalancerStrategy 
> compute the assignment mapping. There is a bug that all the tasks from same 
> job will be assigned to one node, which is not balanced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-635) GenericTaskAssignmentCalculator does not balance task assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504886#comment-15504886
 ] 

ASF GitHub Bot commented on HELIX-635:
--

Github user dasahcc commented on a diff in the pull request:

https://github.com/apache/helix/pull/51#discussion_r79503613
  
--- Diff: helix-core/src/main/java/org/apache/helix/task/TaskConfig.java ---
@@ -184,10 +186,12 @@ public Builder setTargetPartition(String 
targetPartition) {
   return this;
 }
 
+@Deprecated
 public boolean isSuccessOptional() {
--- End diff --

Added to coment


> GenericTaskAssignmentCalculator does not balance task assignment
> 
>
> Key: HELIX-635
> URL: https://issues.apache.org/jira/browse/HELIX-635
> Project: Apache Helix
>  Issue Type: Bug
>  Components: helix-core
>Reporter: Junkai Xue
>Assignee: Junkai Xue
> Fix For: 0.6.x
>
>
> Currently, GenericTaskAssignmentCalculator using AutoRebalancerStrategy 
> compute the assignment mapping. There is a bug that all the tasks from same 
> job will be assigned to one node, which is not balanced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-635) GenericTaskAssignmentCalculator does not balance task assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504854#comment-15504854
 ] 

ASF GitHub Bot commented on HELIX-635:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/51#discussion_r79501230
  
--- Diff: helix-core/src/main/java/org/apache/helix/task/TaskConfig.java ---
@@ -184,10 +186,12 @@ public Builder setTargetPartition(String 
targetPartition) {
   return this;
 }
 
+@Deprecated
 public boolean isSuccessOptional() {
--- End diff --

Add comment says this field is ignored by Helix.


> GenericTaskAssignmentCalculator does not balance task assignment
> 
>
> Key: HELIX-635
> URL: https://issues.apache.org/jira/browse/HELIX-635
> Project: Apache Helix
>  Issue Type: Bug
>  Components: helix-core
>Reporter: Junkai Xue
>Assignee: Junkai Xue
> Fix For: 0.6.x
>
>
> Currently, GenericTaskAssignmentCalculator using AutoRebalancerStrategy 
> compute the assignment mapping. There is a bug that all the tasks from same 
> job will be assigned to one node, which is not balanced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-635) GenericTaskAssignmentCalculator does not balance task assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504849#comment-15504849
 ] 

ASF GitHub Bot commented on HELIX-635:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/51#discussion_r79500989
  
--- Diff: helix-core/src/main/java/org/apache/helix/task/JobConfig.java ---
@@ -449,11 +446,6 @@ public Builder setMaxAttemptsPerTask(int v) {
   return this;
 }
 
-public Builder setMaxForcedReassignmentsPerTask(int v) {
--- End diff --

same here, do not remove the method.  Deprecate it and clearly comment that 
this field will no longer used by Helix.


> GenericTaskAssignmentCalculator does not balance task assignment
> 
>
> Key: HELIX-635
> URL: https://issues.apache.org/jira/browse/HELIX-635
> Project: Apache Helix
>  Issue Type: Bug
>  Components: helix-core
>Reporter: Junkai Xue
>Assignee: Junkai Xue
> Fix For: 0.6.x
>
>
> Currently, GenericTaskAssignmentCalculator using AutoRebalancerStrategy 
> compute the assignment mapping. There is a bug that all the tasks from same 
> job will be assigned to one node, which is not balanced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HELIX-635) GenericTaskAssignmentCalculator does not balance task assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488751#comment-15488751
 ] 

ASF GitHub Bot commented on HELIX-635:
--

GitHub user dasahcc opened a pull request:

https://github.com/apache/helix/pull/51

[HELIX-635] GenericTaskAssignmentCalculator does not balance task assignment

Currently, GenericTaskAssignmentCalculator using AutoRebalancerStrategy 
compute the assignment mapping. There is a bug that all the tasks from same job 
will be assigned to one node, which is not balanced.

This fix contains:
1. Implement consistent hashing mapping calculation for 
GenericTaskAssignmentCalculator
2. Remove reassign logics and applied in consistent hashing
3. Add tests for GenericTaskAssignmentCalculator
4. Refactoring TaskAssignmentCalculator API, since ClusterDataCache is 
too large and not all the contents inside are used.
5. Support identical task initialization with job command and number of 
tasks
6. Remove unused MaxForcedReassignmentPerTask field
7. Refactor logics of failure.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dasahcc/helix helix-0.6.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/51.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #51


commit 0a18726fcad7b8a0fe5e77d7a2c9848b86461ccc
Author: Junkai Xue 
Date:   2016-09-13T22:28:57Z

[HELIX-635] GenericTaskAssignmentCalculator rebalance with consistent 
hashing

1. Implement consistent hashing mapping calculation
2. Remove reassign logics and applied in consistent hashing
3. Add tests for GenericTaskAssignmentCalculator

commit 7bb2a9db2396a00bb9a721634a2432240679c657
Author: Junkai Xue 
Date:   2016-09-13T23:00:08Z

Refactor TaskAssignmentCalculator API

Refactoring TaskAssignmentCalculator API, since ClusterDataCache is too 
large and not all the contents inside are used.

commit 27cb5ecfe5f3b07c5c97534222558702a05e5aae
Author: Junkai Xue 
Date:   2016-09-13T23:01:39Z

Job Config and logic refactoring

1. Support identical task initialization with job command and number of 
tasks
2. Remove unused MaxForcedReassignmentPerTask field
3. Refactor logics of failure.




> GenericTaskAssignmentCalculator does not balance task assignment
> 
>
> Key: HELIX-635
> URL: https://issues.apache.org/jira/browse/HELIX-635
> Project: Apache Helix
>  Issue Type: Bug
>  Components: helix-core
>Reporter: Junkai Xue
>Assignee: Junkai Xue
> Fix For: 0.6.x
>
>
> Currently, GenericTaskAssignmentCalculator using AutoRebalancerStrategy 
> compute the assignment mapping. There is a bug that all the tasks from same 
> job will be assigned to one node, which is not balanced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)