date:20180320

[jira] [Commented] (HELIX-682) Stale message should not prevent controller from rebalancing resource

2018-03-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HELIX-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407337#comment-16407337
 ] 

ASF GitHub Bot commented on HELIX-682:
--

GitHub user zhan849 opened a pull request:

https://github.com/apache/helix/pull/156

[HELIX-682] controller should delete obsolete messages with timeout to 
unblock state transition

This RB contains implementations and tests for controller: during 
MessageGenerationPhase, it checks if the pending message should be cleaned up 
on participant to unblock further state transition:

- If partition's current state is same as message's toState, and the 3sec 
timeout already passed, in this case, it's likely that participant failed to 
delete message and controller should proactively remove the message so further 
rebalance could be unblocked
- If partition's current state is same as message's fromState, this means 
the partition is undergoing state transition or the state transition has not 
started yet, in this case, we do nothing
- If partition's current state is neither message's fromState nor toState 
(almost impossible), this means this message is a problematic one, and it is 
safe to delete it immediately so participant would not undergo an unnecessary 
message handling

Message deletion on controller side is async

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhan849/helix harry/controller-msg-dedup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #156


commit 9f789dee0b17886bd97ebf4cc14e9d867043183d
Author: Harry Zhang 
Date:   2018-03-21T01:47:02Z

[HELIX-682] controller should delete obsolete messages with timeout to 
unblock state transition




> Stale message should not prevent controller from rebalancing resource
> -
>
> Key: HELIX-682
> URL: https://issues.apache.org/jira/browse/HELIX-682
> Project: Apache Helix
>  Issue Type: Bug
>Reporter: Hao Zhang
>Priority: Major
>
> Currently during MessageGenerationPhase, we skip re-balancing when there is 
> pending message. Though we assume that participant will delete messages when 
> they finish the task, there will be cases that when ZK is not stable and 
> participant fail to do so, which will leave message un-deleted and thus block 
> rebalance.
> Ideally on controller side, we should try to delete message as well: if 
> partition's current state is same as message's toState, or there is totally 
> invalid message remaining, controller should try to delete message to unblock 
> rebalancing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] helix pull request #156: [HELIX-682] controller should delete obsolete messa...

2018-03-20 Thread zhan849

GitHub user zhan849 opened a pull request:

https://github.com/apache/helix/pull/156

[HELIX-682] controller should delete obsolete messages with timeout to 
unblock state transition

This RB contains implementations and tests for controller: during 
MessageGenerationPhase, it checks if the pending message should be cleaned up 
on participant to unblock further state transition:

- If partition's current state is same as message's toState, and the 3sec 
timeout already passed, in this case, it's likely that participant failed to 
delete message and controller should proactively remove the message so further 
rebalance could be unblocked
- If partition's current state is same as message's fromState, this means 
the partition is undergoing state transition or the state transition has not 
started yet, in this case, we do nothing
- If partition's current state is neither message's fromState nor toState 
(almost impossible), this means this message is a problematic one, and it is 
safe to delete it immediately so participant would not undergo an unnecessary 
message handling

Message deletion on controller side is async

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhan849/helix harry/controller-msg-dedup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #156


commit 9f789dee0b17886bd97ebf4cc14e9d867043183d
Author: Harry Zhang 
Date:   2018-03-21T01:47:02Z

[HELIX-682] controller should delete obsolete messages with timeout to 
unblock state transition




---

[GitHub] helix pull request #155: Features and improvements to Helix RoutingTableProv...

2018-03-20 Thread lei-xia

GitHub user lei-xia opened a pull request:

https://github.com/apache/helix/pull/155

Features and improvements to Helix RoutingTableProvider.

This PR includes a few of features and improvements to existing Helix's 
RoutingTableProvider interface and implements. There are:

1)   Add to getResources() into RoutingTableProvider.
2)   RoutingTableProvider to support direct aggregating routing information 
from CurrentStates in each liveinstance. When sourceDataType is set as 
CurrentState, RoutingTableProvider will listen on CurrentStateChanges and 
refresh routing table from CurrentStates upon changes.
3)   Add RoutingTableSnapshot class to hold a snapshot of routing table 
information and provide API to return RoutingTableSnapshot from 
RoutingTableProvider.
4)   Support RoutingTableChangeListener in RoutingTableProvider, which will 
be called when there is anything changed in the RoutingTable information.





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lei-xia/helix routingTableChanges

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/155.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #155


commit 03656dcc41dfb735e75b970171e9ea80cfe1c0da
Author: Lei Xia 
Date:   2018-02-08T23:52:01Z

Allow to get all resources from RoutingTableProvider class.

commit f908193355e132e149ad9525f7770a15b8bca513
Author: Lei Xia 
Date:   2018-03-20T22:04:30Z

Change RoutingTableProvider to support direct aggregating routing 
information from CurrentStates in each liveinstance. When sourceDataType is set 
as CurrentState, RoutingTableProvider will listen on CurrentStateChanges and 
refresh routing table from CurrentStates upon changes.

commit d2288d61b32f36b08987d06007b2aead68fa1be4
Author: Lei Xia 
Date:   2018-02-17T07:05:49Z

Including version number in Participant and Controller history, and add 
additional logs.

commit 8709e6015930a691b6a72acad207d68b7365bcc1
Author: Lei Xia 
Date:   2018-02-16T00:12:22Z

Add RoutingTableSnapshot class to hold a snapshot of routing table 
information and provide API to return RoutingTableSnapshot from 
RoutingTableProvider.

commit 85537299d46c2f96130671d93d9118bed4a5492f
Author: Junkai Xue 
Date:   2018-02-28T22:12:31Z

Support RoutingTableChangeListener in RoutingTableProvider. Add test for it.




---

[GitHub] helix issue #151: Tutorial update for task framework new features in 0.8

2018-03-20 Thread lei-xia

Github user lei-xia commented on the issue:

https://github.com/apache/helix/pull/151
  
Could you please rebase to head?


---

helix - Build # 1415 - Still Failing

2018-03-20 Thread Apache Jenkins Server

The Apache Jenkins build system has built helix (build #1415)

Status: Still Failing

Check console output at https://builds.apache.org/job/helix/1415/ to view the 
results.

[GitHub] helix pull request #153: Retrieve cached idealMappings for all Rebalancers i...

2018-03-20 Thread lei-xia

GitHub user lei-xia opened a pull request:

https://github.com/apache/helix/pull/153

Retrieve cached idealMappings for all Rebalancers instead of recomputing it 
during every rebalance round.

Retrieve cached idealMappings for all Rebalancers (AutoRebalancer, 
DelayedRebalancer and CustomRebalancer) for any rebalance strategies. This will 
avoid recompute idealmapping (preference list) if there is no 
idealstate/instanceconfig/LiveInstances changes.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lei-xia/helix master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/153.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #153


commit e2f99412b46f282f626a4734f24459c49bd130e1
Author: Lei Xia 
Date:   2018-02-09T18:28:25Z

Retrieve cached idealMappings for all Rebalancers (AutoRebalancer, 
DelayedRebalancer and CustomRebalancer) for any rebalance strategies. This will 
avoid recompute idealmapping (preference list) if there is no 
idealstate/instanceconfig/LiveInstances changes.




---

[GitHub] helix pull request #150: Refactor ClusterDataCache, break it into small cach...

2018-03-20 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/helix/pull/150


---

[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

2018-03-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406845#comment-16406845
 ] 

ASF GitHub Bot commented on HELIX-674:
--

Github user jiajunwang commented on a diff in the pull request:

https://github.com/apache/helix/pull/145#discussion_r175879486
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/TotalCapacityConstraint.java
 ---
@@ -0,0 +1,79 @@
+package org.apache.helix.controller.rebalancer.constraint;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import 
org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint;
+import 
org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider;
+import 
org.apache.helix.api.rebalancer.constraint.dataprovider.PartitionWeightProvider;
+import org.apache.helix.controller.common.ResourcesStateMap;
+import org.apache.helix.controller.rebalancer.util.ResourceUsageCalculator;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class TotalCapacityConstraint extends 
AbstractRebalanceHardConstraint {
+  private final PartitionWeightProvider _partitionWeightProvider;
+  private final CapacityProvider _capacityProvider;
+  // Use to track any assignments that are proposed during the rebalance 
process.
+  // Note these assignments are not reflected in providers.
+  private final Map _pendingUsage;
+
+  public TotalCapacityConstraint(PartitionWeightProvider 
partitionWeightProvider,
+  CapacityProvider capacityProvider) {
+super();
+_partitionWeightProvider = partitionWeightProvider;
+_capacityProvider = capacityProvider;
+_pendingUsage = new HashMap<>();
+  }
+
+  private boolean validate(String resource, String partition, String 
participant) {
+int usage = _capacityProvider.getParticipantUsage(participant) + 
(_pendingUsage
+.containsKey(participant) ? _pendingUsage.get(participant) : 0);
+return +_partitionWeightProvider.getPartitionWeight(resource, 
partition) + usage
--- End diff --

Typo... Good catch.


> Constraint Based Resource Rebalancer
> 
>
> Key: HELIX-674
> URL: https://issues.apache.org/jira/browse/HELIX-674
> Project: Apache Helix
>  Issue Type: New Feature
>Reporter: Jiajun Wang
>Assignee: Jiajun Wang
>Priority: Major
> Fix For: 0.8.x
>
> Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf
>
>
> Helix rebalancer assigns resources according to different strategies. 
> Recently, we optimize the strategy for evenness and minimize movement. 
> However, the evenness here only applies to partition numbers. Moreover, we've 
> got more requests for customizable rebalancer from our users.
> Take partition weight as an example:
> In reality, partition replicas have different size. We use "partition weight" 
> as an abstraction of the partition size. It can be network traffic usage, 
> disk usage, or any other combined factors.
> Given each partition may have different weights, Helix should be able to 
> assign partition accordingly. So that the distribution would be even 
> regarding the weight.
> In this project, we are planning new rebalancer mechanism that generates 
> resource partition assignment according to a list of "constraints". Current 
> rebalance strategy can be regarded as one kind of constraint. Moving forward, 
> Helix users would be able to extend the constraint interface using their own 
> logic.
> Some init discussions are in progress and we will have a proposal posted here 
> soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-20 Thread jiajunwang

Github user jiajunwang commented on a diff in the pull request:

https://github.com/apache/helix/pull/145#discussion_r175879053
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceSoftConstraint.java
 ---
@@ -0,0 +1,56 @@
+package org.apache.helix.api.rebalancer.constraint;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.helix.controller.common.ResourcesStateMap;
+
+import java.util.Map;
+
+public abstract class AbstractRebalanceSoftConstraint {
+  private static int DEFAULT_IMPORTANCE = 1;
+  protected int _importance = DEFAULT_IMPORTANCE;
+
+  /**
+   * Evaluate how the given assignment fits the constraint.
+   * @param resource Target resource
+   * @param proposedAssignment Map of 
+   * @return Evaluation about the assignment. Larger number means better 
fit under this constraint.
+   */
+  public abstract Map evaluate(String resource,
+  Map proposedAssignment);
+
+  /**
+   * @return The soft constraint's importance that will be used to compare 
with other soft constraint results.
+   * Aggregated evaluation score = SUM(constraint_evaluation * importance).
+   */
+  public int getConstraintImportance() {
--- End diff --

It's basically weight of the constraint. However, we already have a weight 
for resource quota.
So avoid using the same word here.


---

[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

2018-03-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406841#comment-16406841
 ] 

ASF GitHub Bot commented on HELIX-674:
--

Github user jiajunwang commented on a diff in the pull request:

https://github.com/apache/helix/pull/145#discussion_r175879053
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceSoftConstraint.java
 ---
@@ -0,0 +1,56 @@
+package org.apache.helix.api.rebalancer.constraint;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.helix.controller.common.ResourcesStateMap;
+
+import java.util.Map;
+
+public abstract class AbstractRebalanceSoftConstraint {
+  private static int DEFAULT_IMPORTANCE = 1;
+  protected int _importance = DEFAULT_IMPORTANCE;
+
+  /**
+   * Evaluate how the given assignment fits the constraint.
+   * @param resource Target resource
+   * @param proposedAssignment Map of 
+   * @return Evaluation about the assignment. Larger number means better 
fit under this constraint.
+   */
+  public abstract Map evaluate(String resource,
+  Map proposedAssignment);
+
+  /**
+   * @return The soft constraint's importance that will be used to compare 
with other soft constraint results.
+   * Aggregated evaluation score = SUM(constraint_evaluation * importance).
+   */
+  public int getConstraintImportance() {
--- End diff --

It's basically weight of the constraint. However, we already have a weight 
for resource quota.
So avoid using the same word here.


> Constraint Based Resource Rebalancer
> 
>
> Key: HELIX-674
> URL: https://issues.apache.org/jira/browse/HELIX-674
> Project: Apache Helix
>  Issue Type: New Feature
>Reporter: Jiajun Wang
>Assignee: Jiajun Wang
>Priority: Major
> Fix For: 0.8.x
>
> Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf
>
>
> Helix rebalancer assigns resources according to different strategies. 
> Recently, we optimize the strategy for evenness and minimize movement. 
> However, the evenness here only applies to partition numbers. Moreover, we've 
> got more requests for customizable rebalancer from our users.
> Take partition weight as an example:
> In reality, partition replicas have different size. We use "partition weight" 
> as an abstraction of the partition size. It can be network traffic usage, 
> disk usage, or any other combined factors.
> Given each partition may have different weights, Helix should be able to 
> assign partition accordingly. So that the distribution would be even 
> regarding the weight.
> In this project, we are planning new rebalancer mechanism that generates 
> resource partition assignment according to a list of "constraints". Current 
> rebalance strategy can be regarded as one kind of constraint. Moving forward, 
> Helix users would be able to extend the constraint interface using their own 
> logic.
> Some init discussions are in progress and we will have a proposal posted here 
> soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

2018-03-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406748#comment-16406748
 ] 

ASF GitHub Bot commented on HELIX-674:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/145#discussion_r175833400
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceSoftConstraint.java
 ---
@@ -0,0 +1,56 @@
+package org.apache.helix.api.rebalancer.constraint;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.helix.controller.common.ResourcesStateMap;
+
+import java.util.Map;
+
+public abstract class AbstractRebalanceSoftConstraint {
+  private static int DEFAULT_IMPORTANCE = 1;
+  protected int _importance = DEFAULT_IMPORTANCE;
+
+  /**
+   * Evaluate how the given assignment fits the constraint.
+   * @param resource Target resource
+   * @param proposedAssignment Map of 
+   * @return Evaluation about the assignment. Larger number means better 
fit under this constraint.
+   */
+  public abstract Map evaluate(String resource,
+  Map proposedAssignment);
+
+  /**
+   * @return The soft constraint's importance that will be used to compare 
with other soft constraint results.
+   * Aggregated evaluation score = SUM(constraint_evaluation * importance).
+   */
+  public int getConstraintImportance() {
--- End diff --

Is importance same as priority?


> Constraint Based Resource Rebalancer
> 
>
> Key: HELIX-674
> URL: https://issues.apache.org/jira/browse/HELIX-674
> Project: Apache Helix
>  Issue Type: New Feature
>Reporter: Jiajun Wang
>Assignee: Jiajun Wang
>Priority: Major
> Fix For: 0.8.x
>
> Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf
>
>
> Helix rebalancer assigns resources according to different strategies. 
> Recently, we optimize the strategy for evenness and minimize movement. 
> However, the evenness here only applies to partition numbers. Moreover, we've 
> got more requests for customizable rebalancer from our users.
> Take partition weight as an example:
> In reality, partition replicas have different size. We use "partition weight" 
> as an abstraction of the partition size. It can be network traffic usage, 
> disk usage, or any other combined factors.
> Given each partition may have different weights, Helix should be able to 
> assign partition accordingly. So that the distribution would be even 
> regarding the weight.
> In this project, we are planning new rebalancer mechanism that generates 
> resource partition assignment according to a list of "constraints". Current 
> rebalance strategy can be regarded as one kind of constraint. Moving forward, 
> Helix users would be able to extend the constraint interface using their own 
> logic.
> Some init discussions are in progress and we will have a proposal posted here 
> soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

2018-03-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406750#comment-16406750
 ] 

ASF GitHub Bot commented on HELIX-674:
--

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/145#discussion_r175858026
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/TotalCapacityConstraint.java
 ---
@@ -0,0 +1,79 @@
+package org.apache.helix.controller.rebalancer.constraint;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import 
org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint;
+import 
org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider;
+import 
org.apache.helix.api.rebalancer.constraint.dataprovider.PartitionWeightProvider;
+import org.apache.helix.controller.common.ResourcesStateMap;
+import org.apache.helix.controller.rebalancer.util.ResourceUsageCalculator;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class TotalCapacityConstraint extends 
AbstractRebalanceHardConstraint {
+  private final PartitionWeightProvider _partitionWeightProvider;
+  private final CapacityProvider _capacityProvider;
+  // Use to track any assignments that are proposed during the rebalance 
process.
+  // Note these assignments are not reflected in providers.
+  private final Map _pendingUsage;
+
+  public TotalCapacityConstraint(PartitionWeightProvider 
partitionWeightProvider,
+  CapacityProvider capacityProvider) {
+super();
+_partitionWeightProvider = partitionWeightProvider;
+_capacityProvider = capacityProvider;
+_pendingUsage = new HashMap<>();
+  }
+
+  private boolean validate(String resource, String partition, String 
participant) {
+int usage = _capacityProvider.getParticipantUsage(participant) + 
(_pendingUsage
+.containsKey(participant) ? _pendingUsage.get(participant) : 0);
+return +_partitionWeightProvider.getPartitionWeight(resource, 
partition) + usage
--- End diff --

why there is a "+" in begining of the statement?


> Constraint Based Resource Rebalancer
> 
>
> Key: HELIX-674
> URL: https://issues.apache.org/jira/browse/HELIX-674
> Project: Apache Helix
>  Issue Type: New Feature
>Reporter: Jiajun Wang
>Assignee: Jiajun Wang
>Priority: Major
> Fix For: 0.8.x
>
> Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf
>
>
> Helix rebalancer assigns resources according to different strategies. 
> Recently, we optimize the strategy for evenness and minimize movement. 
> However, the evenness here only applies to partition numbers. Moreover, we've 
> got more requests for customizable rebalancer from our users.
> Take partition weight as an example:
> In reality, partition replicas have different size. We use "partition weight" 
> as an abstraction of the partition size. It can be network traffic usage, 
> disk usage, or any other combined factors.
> Given each partition may have different weights, Helix should be able to 
> assign partition accordingly. So that the distribution would be even 
> regarding the weight.
> In this project, we are planning new rebalancer mechanism that generates 
> resource partition assignment according to a list of "constraints". Current 
> rebalance strategy can be regarded as one kind of constraint. Moving forward, 
> Helix users would be able to extend the constraint interface using their own 
> logic.
> Some init discussions are in progress and we will have a proposal posted here 
> soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-20 Thread lei-xia

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/145#discussion_r175858026
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/TotalCapacityConstraint.java
 ---
@@ -0,0 +1,79 @@
+package org.apache.helix.controller.rebalancer.constraint;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import 
org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint;
+import 
org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider;
+import 
org.apache.helix.api.rebalancer.constraint.dataprovider.PartitionWeightProvider;
+import org.apache.helix.controller.common.ResourcesStateMap;
+import org.apache.helix.controller.rebalancer.util.ResourceUsageCalculator;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class TotalCapacityConstraint extends 
AbstractRebalanceHardConstraint {
+  private final PartitionWeightProvider _partitionWeightProvider;
+  private final CapacityProvider _capacityProvider;
+  // Use to track any assignments that are proposed during the rebalance 
process.
+  // Note these assignments are not reflected in providers.
+  private final Map _pendingUsage;
+
+  public TotalCapacityConstraint(PartitionWeightProvider 
partitionWeightProvider,
+  CapacityProvider capacityProvider) {
+super();
+_partitionWeightProvider = partitionWeightProvider;
+_capacityProvider = capacityProvider;
+_pendingUsage = new HashMap<>();
+  }
+
+  private boolean validate(String resource, String partition, String 
participant) {
+int usage = _capacityProvider.getParticipantUsage(participant) + 
(_pendingUsage
+.containsKey(participant) ? _pendingUsage.get(participant) : 0);
+return +_partitionWeightProvider.getPartitionWeight(resource, 
partition) + usage
--- End diff --

why there is a "+" in begining of the statement?


---

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-20 Thread lei-xia

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/145#discussion_r175833400
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceSoftConstraint.java
 ---
@@ -0,0 +1,56 @@
+package org.apache.helix.api.rebalancer.constraint;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.helix.controller.common.ResourcesStateMap;
+
+import java.util.Map;
+
+public abstract class AbstractRebalanceSoftConstraint {
+  private static int DEFAULT_IMPORTANCE = 1;
+  protected int _importance = DEFAULT_IMPORTANCE;
+
+  /**
+   * Evaluate how the given assignment fits the constraint.
+   * @param resource Target resource
+   * @param proposedAssignment Map of 
+   * @return Evaluation about the assignment. Larger number means better 
fit under this constraint.
+   */
+  public abstract Map evaluate(String resource,
+  Map proposedAssignment);
+
+  /**
+   * @return The soft constraint's importance that will be used to compare 
with other soft constraint results.
+   * Aggregated evaluation score = SUM(constraint_evaluation * importance).
+   */
+  public int getConstraintImportance() {
--- End diff --

Is importance same as priority?


---

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-20 Thread lei-xia

Github user lei-xia commented on a diff in the pull request:

https://github.com/apache/helix/pull/145#discussion_r175833652
  
--- Diff: 
helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceSoftConstraint.java
 ---
@@ -0,0 +1,56 @@
+package org.apache.helix.api.rebalancer.constraint;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.helix.controller.common.ResourcesStateMap;
+
+import java.util.Map;
+
+public abstract class AbstractRebalanceSoftConstraint {
+  private static int DEFAULT_IMPORTANCE = 1;
+  protected int _importance = DEFAULT_IMPORTANCE;
+
+  /**
+   * Evaluate how the given assignment fits the constraint.
+   * @param resource Target resource
+   * @param proposedAssignment Map of 
+   * @return Evaluation about the assignment. Larger number means better 
fit under this constraint.
+   */
+  public abstract Map evaluate(String resource,
+  Map proposedAssignment);
+
+  /**
+   * @return The soft constraint's importance that will be used to compare 
with other soft constraint results.
+   * Aggregated evaluation score = SUM(constraint_evaluation * importance).
+   */
+  public int getConstraintImportance() {
--- End diff --

And the larger means more important?


---

[jira] [Commented] (HELIX-682) Stale message should not prevent controller from rebalancing resource

[GitHub] helix pull request #156: [HELIX-682] controller should delete obsolete messa...

[GitHub] helix pull request #155: Features and improvements to Helix RoutingTableProv...

[GitHub] helix issue #151: Tutorial update for task framework new features in 0.8

helix - Build # 1415 - Still Failing

[GitHub] helix pull request #153: Retrieve cached idealMappings for all Rebalancers i...

[GitHub] helix pull request #150: Refactor ClusterDataCache, break it into small cach...

[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

15 matches

Site Navigation

Mail list logo

Footer information