[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449080#comment-16449080 ] Hudson commented on HELIX-674: -- FAILURE: Integrated in Jenkins build helix #1448 (See [https://builds.apache.org/job/helix/1448/]) [HELIX-674] Introducing constraints based rebalancing mechanism. (ericwang1985: rev 3d2d57b05af443dcc8658f8b7bfc1fc0d18cd196) * (add) helix-core/src/test/java/org/apache/helix/controller/rebalancer/constraint/dataprovider/MockPartitionWeightProvider.java * (add) helix-core/src/main/java/org/apache/helix/controller/rebalancer/strategy/ConstraintRebalanceStrategy.java * (add) helix-core/src/test/java/org/apache/helix/controller/rebalancer/TestConstraintRebalanceStrategy.java * (add) helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/CapacityProvider.java * (edit) helix-core/src/main/java/org/apache/helix/controller/rebalancer/strategy/crushMapping/CardDealingAdjustmentAlgorithm.java * (add) helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/dataprovider/ZkBasedCapacityProvider.java * (add) helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/PartitionWeightProvider.java * (add) helix-core/src/test/java/org/apache/helix/integration/TestWeightBasedRebalanceUtil.java * (edit) helix-core/src/main/java/org/apache/helix/controller/rebalancer/strategy/AbstractEvenDistributionRebalanceStrategy.java * (add) helix-core/src/main/java/org/apache/helix/examples/WeightAwareRebalanceUtilExample.java * (add) helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/dataprovider/ZkBasedPartitionWeightProvider.java * (add) helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceHardConstraint.java * (add) helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/TotalCapacityConstraint.java * (add) helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java * (add) helix-core/src/test/java/org/apache/helix/controller/rebalancer/constraint/dataprovider/MockCapacityProvider.java * (add) helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceSoftConstraint.java * (add) helix-core/src/main/java/org/apache/helix/controller/rebalancer/util/ResourceUsageCalculator.java * (add) helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/PartitionWeightAwareEvennessConstraint.java > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449076#comment-16449076 ] ASF GitHub Bot commented on HELIX-674: -- Github user asfgit closed the pull request at: https://github.com/apache/helix/pull/145 > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448868#comment-16448868 ] ASF GitHub Bot commented on HELIX-674: -- Github user lei-xia commented on the issue: https://github.com/apache/helix/pull/145 Can you rebase to HEAD? > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409889#comment-16409889 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r176488707 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceSoftConstraint.java --- @@ -0,0 +1,56 @@ +package org.apache.helix.api.rebalancer.constraint; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.helix.controller.common.ResourcesStateMap; + +import java.util.Map; + +public abstract class AbstractRebalanceSoftConstraint { + private static int DEFAULT_IMPORTANCE = 1; + protected int _importance = DEFAULT_IMPORTANCE; + + /** + * Evaluate how the given assignment fits the constraint. + * @param resource Target resource + * @param proposedAssignment Map of + * @return Evaluation about the assignment. Larger number means better fit under this constraint. --- End diff -- @return score of the assignment? > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409890#comment-16409890 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r176490613 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/CapacityProvider.java --- @@ -0,0 +1,48 @@ +package org.apache.helix.api.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * An interface for getting participant capacity information. + * The return value will be used to estimate available capacity, as well as utilization in percentage for prioritizing participants. + * + * Note that all return values of the provider are supposed to be in the same unit. + * For example, if the provider is for memory capacity of a host (total memory size is 8G, current usage is 512MB), + * the return values could be {@link org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider#getParticipantCapacity(String)} = 8192, and {@link org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider#getParticipantUsage(String)} = 512. + * Another example, if the provider is for partition count capacity (max count 1000 partitions, currently no partition assigned), + * the return values should be {@link org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider#getParticipantCapacity(String)} = 1000, and {@link org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider#getParticipantUsage(String)} = 0. + * + * Moreover, while this provider is used together with a {@link org.apache.helix.api.rebalancer.constraint.dataprovider.PartitionWeightProvider PartitionWeightProvider}, + * both providers are supposed to return values in the same unit so they can be used to estimate the resource usage of an proposed assignment. + */ +public interface CapacityProvider { + + /** + * @param participant + * @return The total participant capacity. + */ + int getParticipantCapacity(String participant); + + /** + * @param participant + * @return The participant usage (provisioned capacity). --- End diff -- The comment "(provisioned capacity)" is confusing. Can we remove it? > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This messag
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409891#comment-16409891 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r176489075 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceSoftConstraint.java --- @@ -0,0 +1,56 @@ +package org.apache.helix.api.rebalancer.constraint; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.helix.controller.common.ResourcesStateMap; + +import java.util.Map; + +public abstract class AbstractRebalanceSoftConstraint { + private static int DEFAULT_IMPORTANCE = 1; + protected int _importance = DEFAULT_IMPORTANCE; + + /** + * Evaluate how the given assignment fits the constraint. + * @param resource Target resource + * @param proposedAssignment Map of + * @return Evaluation about the assignment. Larger number means better fit under this constraint. + */ + public abstract Map evaluate(String resource, + Map proposedAssignment); + + /** + * @return The soft constraint's importance that will be used to compare with other soft constraint results. + * Aggregated evaluation score = SUM(constraint_evaluation * importance). + */ + public int getConstraintImportance() { --- End diff -- Good to add that larger means more important. I am still not fully convinced the importance is the right name for this. Maybe @kishoreg has other ideas? I mentioned importance in a comment because I got confused between the usage of the word weight in various places. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406845#comment-16406845 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r175879486 --- Diff: helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/TotalCapacityConstraint.java --- @@ -0,0 +1,79 @@ +package org.apache.helix.controller.rebalancer.constraint; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider; +import org.apache.helix.api.rebalancer.constraint.dataprovider.PartitionWeightProvider; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.util.ResourceUsageCalculator; + +import java.util.HashMap; +import java.util.Map; + +public class TotalCapacityConstraint extends AbstractRebalanceHardConstraint { + private final PartitionWeightProvider _partitionWeightProvider; + private final CapacityProvider _capacityProvider; + // Use to track any assignments that are proposed during the rebalance process. + // Note these assignments are not reflected in providers. + private final Map _pendingUsage; + + public TotalCapacityConstraint(PartitionWeightProvider partitionWeightProvider, + CapacityProvider capacityProvider) { +super(); +_partitionWeightProvider = partitionWeightProvider; +_capacityProvider = capacityProvider; +_pendingUsage = new HashMap<>(); + } + + private boolean validate(String resource, String partition, String participant) { +int usage = _capacityProvider.getParticipantUsage(participant) + (_pendingUsage +.containsKey(participant) ? _pendingUsage.get(participant) : 0); +return +_partitionWeightProvider.getPartitionWeight(resource, partition) + usage --- End diff -- Typo... Good catch. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406841#comment-16406841 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r175879053 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceSoftConstraint.java --- @@ -0,0 +1,56 @@ +package org.apache.helix.api.rebalancer.constraint; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.helix.controller.common.ResourcesStateMap; + +import java.util.Map; + +public abstract class AbstractRebalanceSoftConstraint { + private static int DEFAULT_IMPORTANCE = 1; + protected int _importance = DEFAULT_IMPORTANCE; + + /** + * Evaluate how the given assignment fits the constraint. + * @param resource Target resource + * @param proposedAssignment Map of + * @return Evaluation about the assignment. Larger number means better fit under this constraint. + */ + public abstract Map evaluate(String resource, + Map proposedAssignment); + + /** + * @return The soft constraint's importance that will be used to compare with other soft constraint results. + * Aggregated evaluation score = SUM(constraint_evaluation * importance). + */ + public int getConstraintImportance() { --- End diff -- It's basically weight of the constraint. However, we already have a weight for resource quota. So avoid using the same word here. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406748#comment-16406748 ] ASF GitHub Bot commented on HELIX-674: -- Github user lei-xia commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r175833400 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceSoftConstraint.java --- @@ -0,0 +1,56 @@ +package org.apache.helix.api.rebalancer.constraint; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.helix.controller.common.ResourcesStateMap; + +import java.util.Map; + +public abstract class AbstractRebalanceSoftConstraint { + private static int DEFAULT_IMPORTANCE = 1; + protected int _importance = DEFAULT_IMPORTANCE; + + /** + * Evaluate how the given assignment fits the constraint. + * @param resource Target resource + * @param proposedAssignment Map of + * @return Evaluation about the assignment. Larger number means better fit under this constraint. + */ + public abstract Map evaluate(String resource, + Map proposedAssignment); + + /** + * @return The soft constraint's importance that will be used to compare with other soft constraint results. + * Aggregated evaluation score = SUM(constraint_evaluation * importance). + */ + public int getConstraintImportance() { --- End diff -- Is importance same as priority? > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406749#comment-16406749 ] ASF GitHub Bot commented on HELIX-674: -- Github user lei-xia commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r175833652 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceSoftConstraint.java --- @@ -0,0 +1,56 @@ +package org.apache.helix.api.rebalancer.constraint; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.helix.controller.common.ResourcesStateMap; + +import java.util.Map; + +public abstract class AbstractRebalanceSoftConstraint { + private static int DEFAULT_IMPORTANCE = 1; + protected int _importance = DEFAULT_IMPORTANCE; + + /** + * Evaluate how the given assignment fits the constraint. + * @param resource Target resource + * @param proposedAssignment Map of + * @return Evaluation about the assignment. Larger number means better fit under this constraint. + */ + public abstract Map evaluate(String resource, + Map proposedAssignment); + + /** + * @return The soft constraint's importance that will be used to compare with other soft constraint results. + * Aggregated evaluation score = SUM(constraint_evaluation * importance). + */ + public int getConstraintImportance() { --- End diff -- And the larger means more important? > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406750#comment-16406750 ] ASF GitHub Bot commented on HELIX-674: -- Github user lei-xia commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r175858026 --- Diff: helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/TotalCapacityConstraint.java --- @@ -0,0 +1,79 @@ +package org.apache.helix.controller.rebalancer.constraint; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider; +import org.apache.helix.api.rebalancer.constraint.dataprovider.PartitionWeightProvider; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.util.ResourceUsageCalculator; + +import java.util.HashMap; +import java.util.Map; + +public class TotalCapacityConstraint extends AbstractRebalanceHardConstraint { + private final PartitionWeightProvider _partitionWeightProvider; + private final CapacityProvider _capacityProvider; + // Use to track any assignments that are proposed during the rebalance process. + // Note these assignments are not reflected in providers. + private final Map _pendingUsage; + + public TotalCapacityConstraint(PartitionWeightProvider partitionWeightProvider, + CapacityProvider capacityProvider) { +super(); +_partitionWeightProvider = partitionWeightProvider; +_capacityProvider = capacityProvider; +_pendingUsage = new HashMap<>(); + } + + private boolean validate(String resource, String partition, String participant) { +int usage = _capacityProvider.getParticipantUsage(participant) + (_pendingUsage +.containsKey(participant) ? _pendingUsage.get(participant) : 0); +return +_partitionWeightProvider.getPartitionWeight(resource, partition) + usage --- End diff -- why there is a "+" in begining of the statement? > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399954#comment-16399954 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on the issue: https://github.com/apache/helix/pull/145 PR code updated. Added examples and a lot of comments for clarifying the usage and API define. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399128#comment-16399128 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174576991 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/CapacityProvider.java --- @@ -0,0 +1,38 @@ +package org.apache.helix.api.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * An interface for getting participant capacity information. + */ +public interface CapacityProvider { + + /** + * @param participant + * @return The total participant capacity. + */ + int getParticipantCapacity(String participant); + + /** + * @param participant + * @return The provisioned capacity. + */ + int getParticipantProvisioned(String participant); --- End diff -- Make sense. Let me add more comments. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399127#comment-16399127 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174576961 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util; + +import org.apache.helix.HelixException; +import org.apache.helix.ZNRecord; +import org.apache.helix.api.config.RebalanceConfig; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.controller.common.PartitionStateMap; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.strategy.ConstraintRebalanceStrategy; +import org.apache.helix.controller.stages.ClusterDataCache; +import org.apache.helix.model.*; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * A rebalance tool that generate an resource partition assignment based on the input. + * Note the assignment won't be automatically applied to the cluster. Applications are supposed to + * apply the change. + */ +public class WeightAwareRebalanceUtil { + private final ClusterConfig _clusterConfig; + private final Map _instanceConfigMap = new HashMap<>(); + // For the possible customized state models. + private final Map _stateModelDefs = new HashMap<>(); + private final ClusterDataCache _dataCache; + + public enum RebalanceOption { +INIT, +REASSIGN + } + + public WeightAwareRebalanceUtil(ClusterConfig clusterConfig, + List instanceConfigs) { +for (InstanceConfig instanceConfig : instanceConfigs) { + _instanceConfigMap.put(instanceConfig.getInstanceName(), instanceConfig); +} +_clusterConfig = clusterConfig; + +_dataCache = new ClusterDataCache(); +_dataCache.setInstanceConfigMap(_instanceConfigMap); +_dataCache.setClusterConfig(_clusterConfig); +List liveInstanceList = new ArrayList<>(); +for (String instance : _instanceConfigMap.keySet()) { + LiveInstance liveInstance = new LiveInstance(instance); + liveInstanceList.add(liveInstance); +} +_dataCache.setLiveInstances(liveInstanceList); + } + + /** + * The method to generate partition assignment mappings. + * + * @param resourceConfigsConfig of all the resources that need to be rebalanced. + * The tool throws Exception if any resource has no IS or broken/uninitialized IS. + * The tool throws Exception if any resource is in full-auto mode. + * @param existingAssignment The existing partition assignment of the input resources. + * @param option INIT or REASSIGN + * INIT: Keep existing assignment. Only generate new partition assignment. --- End diff -- You are right, Subbu. The existingAssignment is old partition assignment. For incremental, those old assignments won't be changed unless they are not valid (compared with State Model requirement). For full rebalance, the old assignments are only used to minimize movement if possible. But all existing assignment will be recalculated. And totally agree with your suggestion that make 2 separate APIs. This also helps clean the javadoc for parameters. I will do it. Thanks for the comment. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399126#comment-16399126 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174576953 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util; + +import org.apache.helix.HelixException; +import org.apache.helix.ZNRecord; +import org.apache.helix.api.config.RebalanceConfig; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.controller.common.PartitionStateMap; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.strategy.ConstraintRebalanceStrategy; +import org.apache.helix.controller.stages.ClusterDataCache; +import org.apache.helix.model.*; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * A rebalance tool that generate an resource partition assignment based on the input. + * Note the assignment won't be automatically applied to the cluster. Applications are supposed to + * apply the change. + */ --- End diff -- I've already created the class, haha. I will start with 3 examples, basic case, use ZK, and topology aware. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398867#comment-16398867 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174526219 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/PartitionQuotaProvider.java --- @@ -0,0 +1,33 @@ +package org.apache.helix.api.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * An interface for getting partition quota information. + */ +public interface PartitionQuotaProvider { + + /** + * @param resource + * @param partition + * @return The quota that is required by a certain partition in the specified resource + */ + int getPartitionQuota(String resource, String partition); --- End diff -- Clarify it with comments int he javadoc, with examples. I prefer weight rather than quota A specific example class with a lot of javadocs on different scenarios and how these classes are to be used, will not hurt. Tests can serve as examples, but they are usually sloppy, and if they are examples, then they tend to be copy-pasted the same way in real code. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398864#comment-16398864 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174525270 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/CapacityProvider.java --- @@ -0,0 +1,38 @@ +package org.apache.helix.api.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * An interface for getting participant capacity information. + */ +public interface CapacityProvider { + + /** + * @param participant + * @return The total participant capacity. + */ + int getParticipantCapacity(String participant); + + /** + * @param participant + * @return The provisioned capacity. + */ + int getParticipantProvisioned(String participant); --- End diff -- Please provide examples in the javadoc. In this case, I would add something like: "For example if the constraint is on memory used by a resource in a participant, and the participant is provisioned with a max of 10MB for resources, out of which 40k is used, the getParticpantUsage() should return 40, and getParticipantCapacity() should return 10. On the other hand, if the constraint is on number of partitions in a participant, and the particpant has been provisioned to for a max of 1200 partitions, but has only 65, then getParticipantUsage() should return 65, and getParticipantCapacity() should return 1200" -- something like this. It does not hurt to have more comments. You will get less questions. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398865#comment-16398865 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174528962 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util; + +import org.apache.helix.HelixException; +import org.apache.helix.ZNRecord; +import org.apache.helix.api.config.RebalanceConfig; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.controller.common.PartitionStateMap; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.strategy.ConstraintRebalanceStrategy; +import org.apache.helix.controller.stages.ClusterDataCache; +import org.apache.helix.model.*; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * A rebalance tool that generate an resource partition assignment based on the input. + * Note the assignment won't be automatically applied to the cluster. Applications are supposed to + * apply the change. + */ +public class WeightAwareRebalanceUtil { + private final ClusterConfig _clusterConfig; + private final Map _instanceConfigMap = new HashMap<>(); + // For the possible customized state models. + private final Map _stateModelDefs = new HashMap<>(); + private final ClusterDataCache _dataCache; + + public enum RebalanceOption { +INIT, +REASSIGN + } + + public WeightAwareRebalanceUtil(ClusterConfig clusterConfig, + List instanceConfigs) { +for (InstanceConfig instanceConfig : instanceConfigs) { + _instanceConfigMap.put(instanceConfig.getInstanceName(), instanceConfig); +} +_clusterConfig = clusterConfig; + +_dataCache = new ClusterDataCache(); +_dataCache.setInstanceConfigMap(_instanceConfigMap); +_dataCache.setClusterConfig(_clusterConfig); +List liveInstanceList = new ArrayList<>(); +for (String instance : _instanceConfigMap.keySet()) { + LiveInstance liveInstance = new LiveInstance(instance); + liveInstanceList.add(liveInstance); +} +_dataCache.setLiveInstances(liveInstanceList); + } + + /** + * The method to generate partition assignment mappings. + * + * @param resourceConfigsConfig of all the resources that need to be rebalanced. + * The tool throws Exception if any resource has no IS or broken/uninitialized IS. + * The tool throws Exception if any resource is in full-auto mode. + * @param existingAssignment The existing partition assignment of the input resources. + * @param option INIT or REASSIGN + * INIT: Keep existing assignment. Only generate new partition assignment. --- End diff -- If you are having difficulty naming the variable, use two different APIs. buildIncrementalRebalanceAssignment() buildFullRebalanceAssignment() Or some such. I still don't understand the difference between the two. I suppose incremental means to calculate idealstate assignment for new partitions, given the old partition assignment. Right? In that case the list of new partitions should be taken as an argument, but it is not. Old partitions are not touched. Am I right? I suppose full rebalance means to re-assign existing partitions around (but no new partitions added). Am I right? In this case, we rebalance the partitions, but minimize movement. Am I right? > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas ha
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398866#comment-16398866 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174527442 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util; + +import org.apache.helix.HelixException; +import org.apache.helix.ZNRecord; +import org.apache.helix.api.config.RebalanceConfig; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.controller.common.PartitionStateMap; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.strategy.ConstraintRebalanceStrategy; +import org.apache.helix.controller.stages.ClusterDataCache; +import org.apache.helix.model.*; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * A rebalance tool that generate an resource partition assignment based on the input. + * Note the assignment won't be automatically applied to the cluster. Applications are supposed to + * apply the change. + */ --- End diff -- I cannot emphasize the importance of an example more. Like I mentioned elsewhere, I would suggest adding a WeightBasedRebalanceExample class, in which you can provide example of a few different scenarios (number of partitions as weight, partitin size as weight, a combination of these two, etc.). Add plenty of javadoc in the example class to demonstrate the usage. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397917#comment-16397917 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174326641 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util; + +import org.apache.helix.HelixException; +import org.apache.helix.ZNRecord; +import org.apache.helix.api.config.RebalanceConfig; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.controller.common.PartitionStateMap; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.strategy.ConstraintRebalanceStrategy; +import org.apache.helix.controller.stages.ClusterDataCache; +import org.apache.helix.model.*; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * A rebalance tool that generate an resource partition assignment based on the input. + * Note the assignment won't be automatically applied to the cluster. Applications are supposed to + * apply the change. + */ +public class WeightAwareRebalanceUtil { + private final ClusterConfig _clusterConfig; + private final Map _instanceConfigMap = new HashMap<>(); + // For the possible customized state models. + private final Map _stateModelDefs = new HashMap<>(); + private final ClusterDataCache _dataCache; + + public enum RebalanceOption { +INIT, +REASSIGN + } + + public WeightAwareRebalanceUtil(ClusterConfig clusterConfig, + List instanceConfigs) { +for (InstanceConfig instanceConfig : instanceConfigs) { + _instanceConfigMap.put(instanceConfig.getInstanceName(), instanceConfig); +} +_clusterConfig = clusterConfig; + +_dataCache = new ClusterDataCache(); +_dataCache.setInstanceConfigMap(_instanceConfigMap); +_dataCache.setClusterConfig(_clusterConfig); +List liveInstanceList = new ArrayList<>(); +for (String instance : _instanceConfigMap.keySet()) { + LiveInstance liveInstance = new LiveInstance(instance); + liveInstanceList.add(liveInstance); +} +_dataCache.setLiveInstances(liveInstanceList); + } + + /** + * The method to generate partition assignment mappings. + * + * @param resourceConfigsConfig of all the resources that need to be rebalanced. + * The tool throws Exception if any resource has no IS or broken/uninitialized IS. + * The tool throws Exception if any resource is in full-auto mode. + * @param existingAssignment The existing partition assignment of the input resources. + * @param option INIT or REASSIGN + * INIT: Keep existing assignment. Only generate new partition assignment. --- End diff -- And about the last point. You are correct. Since the rebalance algorithm is optimized to minimize movement, even with full rebalance, the new mapping is supposed to keep as much same assignment as the old one. Just not guaranteed. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints"
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397912#comment-16397912 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174326382 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util; + +import org.apache.helix.HelixException; +import org.apache.helix.ZNRecord; +import org.apache.helix.api.config.RebalanceConfig; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.controller.common.PartitionStateMap; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.strategy.ConstraintRebalanceStrategy; +import org.apache.helix.controller.stages.ClusterDataCache; +import org.apache.helix.model.*; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * A rebalance tool that generate an resource partition assignment based on the input. + * Note the assignment won't be automatically applied to the cluster. Applications are supposed to + * apply the change. + */ --- End diff -- Good suggestion. Will add it. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397911#comment-16397911 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174326343 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util; + +import org.apache.helix.HelixException; +import org.apache.helix.ZNRecord; +import org.apache.helix.api.config.RebalanceConfig; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.controller.common.PartitionStateMap; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.strategy.ConstraintRebalanceStrategy; +import org.apache.helix.controller.stages.ClusterDataCache; +import org.apache.helix.model.*; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * A rebalance tool that generate an resource partition assignment based on the input. + * Note the assignment won't be automatically applied to the cluster. Applications are supposed to + * apply the change. + */ +public class WeightAwareRebalanceUtil { + private final ClusterConfig _clusterConfig; + private final Map _instanceConfigMap = new HashMap<>(); + // For the possible customized state models. + private final Map _stateModelDefs = new HashMap<>(); + private final ClusterDataCache _dataCache; + + public enum RebalanceOption { +INIT, +REASSIGN + } + + public WeightAwareRebalanceUtil(ClusterConfig clusterConfig, + List instanceConfigs) { +for (InstanceConfig instanceConfig : instanceConfigs) { + _instanceConfigMap.put(instanceConfig.getInstanceName(), instanceConfig); +} +_clusterConfig = clusterConfig; + +_dataCache = new ClusterDataCache(); +_dataCache.setInstanceConfigMap(_instanceConfigMap); +_dataCache.setClusterConfig(_clusterConfig); +List liveInstanceList = new ArrayList<>(); +for (String instance : _instanceConfigMap.keySet()) { + LiveInstance liveInstance = new LiveInstance(instance); + liveInstanceList.add(liveInstance); +} +_dataCache.setLiveInstances(liveInstanceList); + } + + /** + * The method to generate partition assignment mappings. + * + * @param resourceConfigsConfig of all the resources that need to be rebalanced. + * The tool throws Exception if any resource has no IS or broken/uninitialized IS. + * The tool throws Exception if any resource is in full-auto mode. + * @param existingAssignment The existing partition assignment of the input resources. + * @param option INIT or REASSIGN + * INIT: Keep existing assignment. Only generate new partition assignment. --- End diff -- "Mode" is confusing while we already have "RebalanceMode". About the option name, how about "incremental" rebalance & "full" rebalance? I think those 2 names are much better. I will add the comment about the resourceConfig, this is where users should specify the partitions. Also, as a standalone tool, most rebalance options won't work even configured in the cluster config / instance config. I will add comments to point it out. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partiti
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397905#comment-16397905 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174324850 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util; + +import org.apache.helix.HelixException; +import org.apache.helix.ZNRecord; +import org.apache.helix.api.config.RebalanceConfig; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.controller.common.PartitionStateMap; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.strategy.ConstraintRebalanceStrategy; +import org.apache.helix.controller.stages.ClusterDataCache; +import org.apache.helix.model.*; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * A rebalance tool that generate an resource partition assignment based on the input. + * Note the assignment won't be automatically applied to the cluster. Applications are supposed to --- End diff -- Good point. Changed. The tool is simplified version of rebalancer. We are planning a new rebalancer that calculates assignment according to constraints. But there will be much more detail logics for different rebalance requirements. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397898#comment-16397898 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174324068 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/PartitionQuotaProvider.java --- @@ -0,0 +1,33 @@ +package org.apache.helix.api.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * An interface for getting partition quota information. + */ +public interface PartitionQuotaProvider { --- End diff -- Should be PartitionWeightProvider based on previous comments and discussions. Info would be too generic here, participant weight is strictly related to the participant capacity. Given we have more date defined for the constraints, we can have more provider. Constraint interface is not related to these providers. So we can always extend the provider list later. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397892#comment-16397892 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174323138 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/CapacityProvider.java --- @@ -0,0 +1,38 @@ +package org.apache.helix.api.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * An interface for getting participant capacity information. + */ +public interface CapacityProvider { + + /** + * @param participant + * @return The total participant capacity. + */ + int getParticipantCapacity(String participant); + + /** + * @param participant + * @return The provisioned capacity. + */ + int getParticipantProvisioned(String participant); --- End diff -- Let's use getParticipantUsage() > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397893#comment-16397893 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174323213 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/PartitionQuotaProvider.java --- @@ -0,0 +1,33 @@ +package org.apache.helix.api.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * An interface for getting partition quota information. + */ +public interface PartitionQuotaProvider { + + /** + * @param resource + * @param partition + * @return The quota that is required by a certain partition in the specified resource + */ + int getPartitionQuota(String resource, String partition); --- End diff -- It is. I'm struggling to make the choice between weight or quota. I named it quota to avoid confusing with the "constraint weight". Given we named the constraint weight to importance, let's name it to weight here. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397891#comment-16397891 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174323126 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/CapacityProvider.java --- @@ -0,0 +1,38 @@ +package org.apache.helix.api.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * An interface for getting participant capacity information. + */ +public interface CapacityProvider { + + /** + * @param participant + * @return The total participant capacity. + */ + int getParticipantCapacity(String participant); --- End diff -- This API for total capacity. Since we need to calculate the percentage of usage for prioritizing, we need both total capacity and current usage. Given this, the second option looks better. Thanks for the comments. I will make the change. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397894#comment-16397894 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174323268 --- Diff: helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/PartitionWeightAwareEvennessConstraint.java --- @@ -0,0 +1,94 @@ +package org.apache.helix.controller.rebalancer.constraint; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider; +import org.apache.helix.api.rebalancer.constraint.dataprovider.PartitionQuotaProvider; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.util.ResourceUsageCalculator; + +import java.util.HashMap; +import java.util.Map; + +public class PartitionWeightAwareEvennessConstraint extends AbstractRebalanceSoftConstraint { + private final int _weight; + private final PartitionQuotaProvider _partitionQuotaProvider; + private final CapacityProvider _capacityProvider; + // Use to track any assignments that are proposed during the rebalance process. + // Note these assignments are not reflected in providers. + private final Map _pendingUsage; + + public PartitionWeightAwareEvennessConstraint(PartitionQuotaProvider partitionQuotaProvider, + CapacityProvider capacityProvider, int weight) { +_partitionQuotaProvider = partitionQuotaProvider; +_capacityProvider = capacityProvider; +_weight = weight; --- End diff -- You are correct. And Importance is a good name, I think. Let me change it and add some comments. Also moved to the superclass. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397843#comment-16397843 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174308429 --- Diff: helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/PartitionWeightAwareEvennessConstraint.java --- @@ -0,0 +1,94 @@ +package org.apache.helix.controller.rebalancer.constraint; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider; +import org.apache.helix.api.rebalancer.constraint.dataprovider.PartitionQuotaProvider; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.util.ResourceUsageCalculator; + +import java.util.HashMap; +import java.util.Map; + +public class PartitionWeightAwareEvennessConstraint extends AbstractRebalanceSoftConstraint { + private final int _weight; + private final PartitionQuotaProvider _partitionQuotaProvider; + private final CapacityProvider _capacityProvider; + // Use to track any assignments that are proposed during the rebalance process. + // Note these assignments are not reflected in providers. + private final Map _pendingUsage; + + public PartitionWeightAwareEvennessConstraint(PartitionQuotaProvider partitionQuotaProvider, + CapacityProvider capacityProvider, int weight) { +_partitionQuotaProvider = partitionQuotaProvider; --- End diff -- add super() > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397844#comment-16397844 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174310141 --- Diff: helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/dataprovider/ZkBasedCapacityProvider.java --- @@ -0,0 +1,202 @@ +package org.apache.helix.controller.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.helix.*; +import org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider; +import org.apache.helix.manager.zk.ZNRecordSerializer; +import org.apache.helix.store.zk.ZkHelixPropertyStore; + +import java.util.HashMap; +import java.util.Map; + +/** + * A capacity provider based on ZK node. + * This class support persistent through Helix Property Store. + */ +public class ZkBasedCapacityProvider implements CapacityProvider { + public static final int DEFAULT_CAPACITY_VALUE = 0; + private static final String ROOT = "/PARTICIPANT_CAPACITY"; + + private final ZkHelixPropertyStore _propertyStore; + private final String _dimensionPath; + private ParticipantCapacity _capacity; + + /** + * @param propertyStore The store that will be used to persist capacity information. + * @param dimensionName Identify of the capacity attribute. For example memory, CPU. + */ + public ZkBasedCapacityProvider(ZkHelixPropertyStore propertyStore, + String dimensionName) { +_propertyStore = propertyStore; +_dimensionPath = ROOT + "/" + dimensionName; + +ZNRecord existingRecord = _propertyStore.get(_dimensionPath, null, AccessOption.PERSISTENT); +if (existingRecord == null) { + // Create a capacity object using default capacity (DEFAULT_CAPACITY_VALUE). + _capacity = new ParticipantCapacity(dimensionName); +} else { + _capacity = new ParticipantCapacity(existingRecord); +} + } + + /** + * @param zkAddr + * @param clusterName + * @param dimensionName Identify of the capacity attribute. For example memory, CPU. + * Need to match resource quota dimension. + */ + public ZkBasedCapacityProvider(String zkAddr, String clusterName, String dimensionName) { +this(new ZkHelixPropertyStore(zkAddr, new ZNRecordSerializer(), +PropertyPathBuilder.propertyStore(clusterName)), dimensionName); + } + + /** + * Update capacity information. + * + * @param capacityMap + * @param usageMap + * @param defaultCapacity Default total capacity if not specified in the map + */ + public void updateCapacity(Map capacityMap, Map usageMap, + int defaultCapacity) { +for (String participant : capacityMap.keySet()) { + _capacity.setCapacity(participant, capacityMap.get(participant)); +} +for (String participant : usageMap.keySet()) { + _capacity.setUsage(participant, usageMap.get(participant)); +} +_capacity.setDefaultCapacity(defaultCapacity); + } + + /** + * @return True if the capacity information is successfully wrote to ZK. + */ + public boolean persistCapacity() { +if (_capacity.isValid()) { + return _propertyStore.set(_dimensionPath, _capacity.getRecord(), AccessOption.PERSISTENT); +} else { + throw new HelixException("Invalid ParticipantCapacity: " + _capacity.getRecord().toString()); +} + } + + @Override + public int getParticipantCapacity(String participant) { +return _capacity.getCapacity(participant); + } + + @Override + public int getParticipantProvisioned(String participant) {
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397840#comment-16397840 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174308502 --- Diff: helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/TotalCapacityConstraint.java --- @@ -0,0 +1,78 @@ +package org.apache.helix.controller.rebalancer.constraint; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider; +import org.apache.helix.api.rebalancer.constraint.dataprovider.PartitionQuotaProvider; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.util.ResourceUsageCalculator; + +import java.util.HashMap; +import java.util.Map; + +public class TotalCapacityConstraint extends AbstractRebalanceHardConstraint { + private final PartitionQuotaProvider _partitionQuotaProvider; + private final CapacityProvider _capacityProvider; + // Use to track any assignments that are proposed during the rebalance process. + // Note these assignments are not reflected in providers. + private final Map _pendingUsage; + + public TotalCapacityConstraint(PartitionQuotaProvider partitionQuotaProvider, + CapacityProvider capacityProvider) { +_partitionQuotaProvider = partitionQuotaProvider; --- End diff -- add super() > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397837#comment-16397837 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174307432 --- Diff: helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/PartitionWeightAwareEvennessConstraint.java --- @@ -0,0 +1,94 @@ +package org.apache.helix.controller.rebalancer.constraint; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.api.rebalancer.constraint.dataprovider.CapacityProvider; +import org.apache.helix.api.rebalancer.constraint.dataprovider.PartitionQuotaProvider; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.util.ResourceUsageCalculator; + +import java.util.HashMap; +import java.util.Map; + +public class PartitionWeightAwareEvennessConstraint extends AbstractRebalanceSoftConstraint { + private final int _weight; + private final PartitionQuotaProvider _partitionQuotaProvider; + private final CapacityProvider _capacityProvider; + // Use to track any assignments that are proposed during the rebalance process. + // Note these assignments are not reflected in providers. + private final Map _pendingUsage; + + public PartitionWeightAwareEvennessConstraint(PartitionQuotaProvider partitionQuotaProvider, + CapacityProvider capacityProvider, int weight) { +_partitionQuotaProvider = partitionQuotaProvider; +_capacityProvider = capacityProvider; +_weight = weight; --- End diff -- Not clear what this weight means. Does this mean the relative importance of this constraint vs any other? If so, maybe we should consider naming these correctly. we have partition weights and constraint weights Also, _weight is not used in the class? Should it move to the super class? > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397845#comment-16397845 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174313286 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util; + +import org.apache.helix.HelixException; +import org.apache.helix.ZNRecord; +import org.apache.helix.api.config.RebalanceConfig; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.controller.common.PartitionStateMap; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.strategy.ConstraintRebalanceStrategy; +import org.apache.helix.controller.stages.ClusterDataCache; +import org.apache.helix.model.*; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * A rebalance tool that generate an resource partition assignment based on the input. + * Note the assignment won't be automatically applied to the cluster. Applications are supposed to --- End diff -- s/Application/users of this class/ Eventually, helix will use this class I suppose > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397847#comment-16397847 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174316247 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util; + +import org.apache.helix.HelixException; +import org.apache.helix.ZNRecord; +import org.apache.helix.api.config.RebalanceConfig; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.controller.common.PartitionStateMap; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.strategy.ConstraintRebalanceStrategy; +import org.apache.helix.controller.stages.ClusterDataCache; +import org.apache.helix.model.*; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * A rebalance tool that generate an resource partition assignment based on the input. + * Note the assignment won't be automatically applied to the cluster. Applications are supposed to + * apply the change. + */ --- End diff -- Good to have extensive javadoc on how to use this tool. Perhaps you intend users to see the tests, in which case you can point to a test on how to use it. One example with zk and one without will be ideal. Thanks. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397839#comment-16397839 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174307115 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/PartitionQuotaProvider.java --- @@ -0,0 +1,33 @@ +package org.apache.helix.api.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * An interface for getting partition quota information. + */ +public interface PartitionQuotaProvider { + + /** + * @param resource + * @param partition + * @return The quota that is required by a certain partition in the specified resource + */ + int getPartitionQuota(String resource, String partition); --- End diff -- Is this the same as the weight of a partition? If so, can we rename to getPartitionWeight()? Just want to make sure that we are consistent with the naming (e.g. PartitionWeightAwareEvennessConstraint) > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397836#comment-16397836 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r173588207 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/CapacityProvider.java --- @@ -0,0 +1,38 @@ +package org.apache.helix.api.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * An interface for getting participant capacity information. + */ +public interface CapacityProvider { + + /** + * @param participant + * @return The total participant capacity. + */ + int getParticipantCapacity(String participant); --- End diff -- If I understand right, this API is to get the current available capacity in a participant? If so, can we rename it to getAvailableCapacity(String participant). Alternatively (if it helps better in naming) we can name them getParticipantCapacity() and getParticipantUsage() > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397842#comment-16397842 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174314122 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util; + +import org.apache.helix.HelixException; +import org.apache.helix.ZNRecord; +import org.apache.helix.api.config.RebalanceConfig; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.controller.common.PartitionStateMap; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.strategy.ConstraintRebalanceStrategy; +import org.apache.helix.controller.stages.ClusterDataCache; +import org.apache.helix.model.*; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * A rebalance tool that generate an resource partition assignment based on the input. + * Note the assignment won't be automatically applied to the cluster. Applications are supposed to + * apply the change. + */ +public class WeightAwareRebalanceUtil { + private final ClusterConfig _clusterConfig; + private final Map _instanceConfigMap = new HashMap<>(); + // For the possible customized state models. + private final Map _stateModelDefs = new HashMap<>(); + private final ClusterDataCache _dataCache; + + public enum RebalanceOption { +INIT, +REASSIGN + } + + public WeightAwareRebalanceUtil(ClusterConfig clusterConfig, + List instanceConfigs) { +for (InstanceConfig instanceConfig : instanceConfigs) { + _instanceConfigMap.put(instanceConfig.getInstanceName(), instanceConfig); +} +_clusterConfig = clusterConfig; + +_dataCache = new ClusterDataCache(); +_dataCache.setInstanceConfigMap(_instanceConfigMap); +_dataCache.setClusterConfig(_clusterConfig); +List liveInstanceList = new ArrayList<>(); +for (String instance : _instanceConfigMap.keySet()) { + LiveInstance liveInstance = new LiveInstance(instance); + liveInstanceList.add(liveInstance); +} +_dataCache.setLiveInstances(liveInstanceList); + } + + /** + * The method to generate partition assignment mappings. + * + * @param resourceConfigsConfig of all the resources that need to be rebalanced. + * The tool throws Exception if any resource has no IS or broken/uninitialized IS. + * The tool throws Exception if any resource is in full-auto mode. + * @param existingAssignment The existing partition assignment of the input resources. + * @param option INIT or REASSIGN + * INIT: Keep existing assignment. Only generate new partition assignment. --- End diff -- instead of option, can we call it 'mode'? Also, it is not clear where we specify new partitions to be assigned. "INIT" also seems to be misleading. Why not "KEEP"? If "INIT" is specified, then is it true that the existing assignments will not be moved? if "REASSIGN" is specified, is it true that the algorithm will try not to move existing assignments, but may do so? > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397838#comment-16397838 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r173588257 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/CapacityProvider.java --- @@ -0,0 +1,38 @@ +package org.apache.helix.api.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * An interface for getting participant capacity information. + */ +public interface CapacityProvider { + + /** + * @param participant + * @return The total participant capacity. + */ + int getParticipantCapacity(String participant); + + /** + * @param participant + * @return The provisioned capacity. + */ + int getParticipantProvisioned(String participant); --- End diff -- Correspondingly, getProvisionedCapacity(String participant) > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397846#comment-16397846 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174313827 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util; + +import org.apache.helix.HelixException; +import org.apache.helix.ZNRecord; +import org.apache.helix.api.config.RebalanceConfig; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceHardConstraint; +import org.apache.helix.api.rebalancer.constraint.AbstractRebalanceSoftConstraint; +import org.apache.helix.controller.common.PartitionStateMap; +import org.apache.helix.controller.common.ResourcesStateMap; +import org.apache.helix.controller.rebalancer.strategy.ConstraintRebalanceStrategy; +import org.apache.helix.controller.stages.ClusterDataCache; +import org.apache.helix.model.*; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * A rebalance tool that generate an resource partition assignment based on the input. + * Note the assignment won't be automatically applied to the cluster. Applications are supposed to + * apply the change. + */ +public class WeightAwareRebalanceUtil { + private final ClusterConfig _clusterConfig; + private final Map _instanceConfigMap = new HashMap<>(); + // For the possible customized state models. + private final Map _stateModelDefs = new HashMap<>(); + private final ClusterDataCache _dataCache; + + public enum RebalanceOption { +INIT, +REASSIGN + } + + public WeightAwareRebalanceUtil(ClusterConfig clusterConfig, + List instanceConfigs) { +for (InstanceConfig instanceConfig : instanceConfigs) { + _instanceConfigMap.put(instanceConfig.getInstanceName(), instanceConfig); +} +_clusterConfig = clusterConfig; + +_dataCache = new ClusterDataCache(); +_dataCache.setInstanceConfigMap(_instanceConfigMap); +_dataCache.setClusterConfig(_clusterConfig); +List liveInstanceList = new ArrayList<>(); +for (String instance : _instanceConfigMap.keySet()) { + LiveInstance liveInstance = new LiveInstance(instance); + liveInstanceList.add(liveInstance); +} +_dataCache.setLiveInstances(liveInstanceList); + } + + /** + * The method to generate partition assignment mappings. + * + * @param resourceConfigsConfig of all the resources that need to be rebalanced. + * The tool throws Exception if any resource has no IS or broken/uninitialized IS. + * The tool throws Exception if any resource is in full-auto mode. + * @param existingAssignment The existing partition assignment of the input resources. --- End diff -- s/input resources/resources specified in param resourceConfigs/ > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397841#comment-16397841 ] ASF GitHub Bot commented on HELIX-674: -- Github user mcvsubbu commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174312501 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/PartitionQuotaProvider.java --- @@ -0,0 +1,33 @@ +package org.apache.helix.api.rebalancer.constraint.dataprovider; + +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/** + * An interface for getting partition quota information. + */ +public interface PartitionQuotaProvider { --- End diff -- if this is providing the weight of a partition, can we call it PartitionWeightProvider? Or, more generic, PartitionInfoProvider (and you can add other info if you need to) > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392232#comment-16392232 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on the issue: https://github.com/apache/helix/pull/137 Changed to another branch instead of using master. https://github.com/apache/helix/pull/145 > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392231#comment-16392231 ] ASF GitHub Bot commented on HELIX-674: -- GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/145 [HELIX-674] Introducing constraints based rebalancing mechanism. Constraint can be customized by application to restrict how rebalancing is processed. The change also includes examples to demonstrate how constraints can be defined and used. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jiajunwang/helix constraint Alternatively you can review and apply these changes as the patch at: https://github.com/apache/helix/pull/145.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #145 commit 195ca9818617e3909c8bf285912778b711edd67c Author: jiajunwang Date: 2018-02-27T01:21:48Z [HELIX-674] Introducing constraints based rebalancing mechanism. Constraint can be customized by application to restrict how rebalancing is processed. The change also includes examples to demonstrate how constraints can be defined and used. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392233#comment-16392233 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang closed the pull request at: https://github.com/apache/helix/pull/137 > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392019#comment-16392019 ] Jiajun Wang commented on HELIX-674: --- Due to lack of Apache Wiki privilege, I cannot create a wiki page for posting design doc there. Upload the design doc as PDF for now. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391771#comment-16391771 ] ASF GitHub Bot commented on HELIX-674: -- Github user jiajunwang commented on the issue: https://github.com/apache/helix/pull/137 @guoyuepeng Thanks for the comment. Fixed. Also updated all TODOs, and refactored the demo features with test cases. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387150#comment-16387150 ] ASF GitHub Bot commented on HELIX-674: -- Github user guoyuepeng commented on the issue: https://github.com/apache/helix/pull/137 MockResourceQuotaProvider#L45 should be if (_partitionQuotaMap.get(resource).containsKey(partition)) { instead of if (_partitionQuotaMap.containsKey(partition)) { could you check it? @jiajunwang > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer
[ https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377882#comment-16377882 ] ASF GitHub Bot commented on HELIX-674: -- GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/137 [HELIX-674] Introducing constraints based rebalancing mechanism. Constraint can be customized by application to restrict how rebalancing is processed. The change also includes examples to demonstrate how constraints can be defined and used. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jiajunwang/helix master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/helix/pull/137.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #137 commit 84052d7acf316619bd284a88b9074aa446548565 Author: jiajunwang Date: 2018-02-27T01:21:48Z Introducing constraints based rebalancing mechanism. Constraint can be customized by application to restrict how rebalancing is processed. The change also includes examples to demonstrate how constraints can be defined and used. > Constraint Based Resource Rebalancer > > > Key: HELIX-674 > URL: https://issues.apache.org/jira/browse/HELIX-674 > Project: Apache Helix > Issue Type: New Feature >Reporter: Jiajun Wang >Assignee: Jiajun Wang >Priority: Major > Fix For: 0.8.x > > > Helix rebalancer assigns resources according to different strategies. > Recently, we optimize the strategy for evenness and minimize movement. > However, the evenness here only applies to partition numbers. Moreover, we've > got more requests for customizable rebalancer from our users. > Take partition weight as an example: > In reality, partition replicas have different size. We use "partition weight" > as an abstraction of the partition size. It can be network traffic usage, > disk usage, or any other combined factors. > Given each partition may have different weights, Helix should be able to > assign partition accordingly. So that the distribution would be even > regarding the weight. > In this project, we are planning new rebalancer mechanism that generates > resource partition assignment according to a list of "constraints". Current > rebalance strategy can be regarded as one kind of constraint. Moving forward, > Helix users would be able to extend the constraint interface using their own > logic. > Some init discussions are in progress and we will have a proposal posted here > soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)