[GitHub] helix issue #300: Daemonize SubscribeChangeEventProcessor thread in Callback...

2019-02-05 Thread jiajunwang
Github user jiajunwang commented on the issue: https://github.com/apache/helix/pull/300 I don't think we can simply make this thread daemon. Actually, it only makes sense when all the handlers are finalized. If one or more handlers are still active, the re-subscribe function

[GitHub] helix pull request #297: ZkClient related improvments

2019-01-25 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/297#discussion_r251134740 --- Diff: helix-core/src/main/java/org/apache/helix/manager/zk/zookeeper/ZkConnection.java --- @@ -84,6 +83,25 @@ public void close() throws

[GitHub] helix issue #297: ZkClient related improvments

2019-01-24 Thread jiajunwang
Github user jiajunwang commented on the issue: https://github.com/apache/helix/pull/297 I split the ZkConnection commit to separate logic change and file movement. Just to ensure the history of change is clear. ---

[GitHub] helix pull request #297: ZkClient related improvments

2019-01-11 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/297 ZkClient related improvments We identify 2 potential issues that may cause a retrying ZK operation failed unexpectedly. These commits fix the problem. You can merge this pull request into a Git

[GitHub] helix pull request #279: Check in the intermediate state calculate stage for...

2018-10-30 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/279 Check in the intermediate state calculate stage for best possible state. Resource rebalance pipeline should continuously processing resource even some resources cannot be calculated

[GitHub] helix pull request #253: Sync the local change about RoutingTableProvider an...

2018-07-17 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/253 Sync the local change about RoutingTableProvider and related metrics change. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jiajunwang/helix

[GitHub] helix pull request #242: Adding more metrics into ClusterStatus about aggreg...

2018-07-13 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/242 Adding more metrics into ClusterStatus about aggregated partition info Also, include a ZK client change that fixes several potential issues we noticed. You can merge this pull request into a Git

[GitHub] helix pull request #237: ZkClient related fix and DelayedAutoRebalancer fix.

2018-07-12 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/237 ZkClient related fix and DelayedAutoRebalancer fix. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jiajunwang/helix master Alternatively

[GitHub] helix pull request #233: [HELIX-726][HELIX-727] Helix monitor metrics improv...

2018-07-10 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/233 [HELIX-726][HELIX-727] Helix monitor metrics improvement Contains 2 changes that improve Helix monitoring. You can merge this pull request into a Git repository by running: $ git pull https

[GitHub] helix pull request #212: [HELIX-] Separate CardDealingAdjustmentAlgorithm fo...

2018-07-09 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/212 [HELIX-] Separate CardDealingAdjustmentAlgorithm for constraint based rebalanc… …e strategy. For keeping CRUSHed stable, we decide to split the logic changes made for constraint

[GitHub] helix pull request #207: [HELIX-707] Fix topstate handoff metrics.

2018-06-26 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/207 [HELIX-707] Fix topstate handoff metrics. We've confirmed a bug in the logic that calculates topstate handoff duration. With this issue, if the previous master instance is offline, an older

[GitHub] helix issue #199: Adding monitoring metrics document.

2018-04-25 Thread jiajunwang
Github user jiajunwang commented on the issue: https://github.com/apache/helix/pull/199 Snapshorts ![index](https://user-images.githubusercontent.com/1803880/39263645-ab6e7bd4-4877-11e8-9e02-23aca0e42986.png) [Apache Helix - Helix Monitoring Metrics.pdf](https

[GitHub] helix pull request #199: Adding monitoring metrics document.

2018-04-25 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/199 Adding monitoring metrics document. Adding monitoring metrics document. Link task framework related sections to the tutorial. You can merge this pull request into a Git repository by running

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-20 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r175879053 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/AbstractRebalanceSoftConstraint.java --- @@ -0,0 +1,56 @@ +package

[GitHub] helix pull request #151: Tutorial update for task framework new features in ...

2018-03-19 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/151 Tutorial update for task framework new features in 0.8 You can merge this pull request into a Git repository by running: $ git pull https://github.com/jiajunwang/helix tutorial

[GitHub] helix issue #145: [HELIX-674] Introducing constraints based rebalancing mech...

2018-03-15 Thread jiajunwang
Github user jiajunwang commented on the issue: https://github.com/apache/helix/pull/145 PR code updated. Added examples and a lot of comments for clarifying the usage and API define. ---

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-14 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174576991 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/CapacityProvider.java --- @@ -0,0 +1,38 @@ +package

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-14 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174576953 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-14 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174576961 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-13 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174326641 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-13 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174326343 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-13 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174326382 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-13 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174324850 --- Diff: helix-core/src/main/java/org/apache/helix/util/WeightAwareRebalanceUtil.java --- @@ -0,0 +1,141 @@ +package org.apache.helix.util

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-13 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174324068 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/PartitionQuotaProvider.java --- @@ -0,0 +1,33 @@ +package

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-13 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174323268 --- Diff: helix-core/src/main/java/org/apache/helix/controller/rebalancer/constraint/PartitionWeightAwareEvennessConstraint.java --- @@ -0,0 +1,94

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-13 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174323126 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/CapacityProvider.java --- @@ -0,0 +1,38 @@ +package

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-13 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174323213 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/PartitionQuotaProvider.java --- @@ -0,0 +1,33 @@ +package

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-13 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/145#discussion_r174323138 --- Diff: helix-core/src/main/java/org/apache/helix/api/rebalancer/constraint/dataprovider/CapacityProvider.java --- @@ -0,0 +1,38 @@ +package

[GitHub] helix pull request #147: [Helix 678] Clear controller event queue when it is...

2018-03-13 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/147 [Helix 678] Clear controller event queue when it is shutdown or no longer the leader Clear controller event queue when it is shutdown or no longer the leader You can merge this pull request

[GitHub] helix issue #137: [HELIX-674] Introducing constraints based rebalancing mech...

2018-03-08 Thread jiajunwang
Github user jiajunwang commented on the issue: https://github.com/apache/helix/pull/137 Changed to another branch instead of using master. https://github.com/apache/helix/pull/145 ---

[GitHub] helix pull request #137: [HELIX-674] Introducing constraints based rebalanci...

2018-03-08 Thread jiajunwang
Github user jiajunwang closed the pull request at: https://github.com/apache/helix/pull/137 ---

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

2018-03-08 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/145 [HELIX-674] Introducing constraints based rebalancing mechanism. Constraint can be customized by application to restrict how rebalancing is processed. The change also includes examples

[GitHub] helix pull request #144: [HELIX-677] Change error log to info/warning when r...

2018-03-08 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/144 [HELIX-677] Change error log to info/warning when replica count canno… …t be read in ResourceMonitor. Some string such as ALL_LIVEINSTANCES in the replica field cannot be processed

[GitHub] helix pull request #143: [HELIX-676] Fix the issue that the controller keep ...

2018-03-08 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/143 [HELIX-676] Fix the issue that the controller keep updating idealstates when there is no real diff. The causes of the problem are that: 1.A previous issue introduced

[GitHub] helix pull request #142: [HELIX-675] Refactor controller start/cleanup logic...

2018-03-08 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/142 [HELIX-675] Refactor controller start/cleanup logic to ensure monitor… … register/reset is handled in any event orders. Due to different possible event process order, controller init

[GitHub] helix issue #137: [HELIX-674] Introducing constraints based rebalancing mech...

2018-03-08 Thread jiajunwang
Github user jiajunwang commented on the issue: https://github.com/apache/helix/pull/137 @guoyuepeng Thanks for the comment. Fixed. Also updated all TODOs, and refactored the demo features with test cases. ---

[GitHub] helix pull request #137: [HELIX-674] Introducing constraints based rebalanci...

2018-02-26 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/137 [HELIX-674] Introducing constraints based rebalancing mechanism. Constraint can be customized by application to restrict how rebalancing is processed. The change also includes examples

[GitHub] helix pull request #131: Bump up ZOOKEEPER version to 3.4.11.

2018-01-25 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/131 Bump up ZOOKEEPER version to 3.4.11. There is a zk connection related bug (ZOOKEEPER-2775) fixed in 3.4.11. Bump up version to get the fix. You can merge this pull request into a Git repository

[GitHub] helix pull request #101: Add task execution throttling manual into TaskFrame...

2017-06-23 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/101 Add task execution throttling manual into TaskFramework document. Add task execution throttling manual into TaskFramework document. Also update version to enable "mvn site". You

[GitHub] helix pull request #89: [HELIX-655] Helix per-participant concurrent task th...

2017-06-23 Thread jiajunwang
Github user jiajunwang closed the pull request at: https://github.com/apache/helix/pull/89 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] helix issue #89: [HELIX-655] Helix per-participant concurrent task throttlin...

2017-06-23 Thread jiajunwang
Github user jiajunwang commented on the issue: https://github.com/apache/helix/pull/89 Closed since master branch updated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] helix pull request #89: [HELIX-655] Helix per-participant concurrent task th...

2017-05-31 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/89#discussion_r119465719 --- Diff: helix-core/src/main/java/org/apache/helix/model/InstanceConfig.java --- @@ -408,6 +410,18 @@ public void setInstanceEnabledForPartition(String

[GitHub] helix issue #89: [HELIX-655] Helix per-participant concurrent task throttlin...

2017-05-25 Thread jiajunwang
Github user jiajunwang commented on the issue: https://github.com/apache/helix/pull/89 Rebase to latest change and apply one additional change as suggested by lei-xia. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] helix pull request #95: [HELIX-657] Fix unexpected idealstate overwrite when...

2017-05-24 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/95 [HELIX-657] Fix unexpected idealstate overwrite when persist assignme… …nt is on. 1. Change persist method from set to update in PersistAssignmentStage. The new updater only

[GitHub] helix pull request #88: [HELIX-654] Running task rebalance

2017-05-19 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/88#discussion_r117420922 --- Diff: helix-core/src/main/java/org/apache/helix/task/JobRebalancer.java --- @@ -455,6 +454,44 @@ private ResourceAssignment computeResourceMapping

[GitHub] helix pull request #88: [HELIX-654] Running task rebalance

2017-05-17 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/88#discussion_r117137720 --- Diff: helix-core/src/main/java/org/apache/helix/task/JobRebalancer.java --- @@ -455,6 +454,44 @@ private ResourceAssignment computeResourceMapping

[GitHub] helix pull request #88: [HELIX-654] Running task rebalance

2017-05-17 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/88#discussion_r117135837 --- Diff: helix-core/src/main/java/org/apache/helix/task/JobRebalancer.java --- @@ -420,6 +411,14 @@ private ResourceAssignment computeResourceMapping

[GitHub] helix pull request #89: [HELIX-655] Helix per-participant concurrent task th...

2017-05-15 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/89#discussion_r116576058 --- Diff: helix-core/src/main/java/org/apache/helix/controller/stages/CurrentStateOutput.java --- @@ -260,13 +260,72 @@ public Message getPendingState

[GitHub] helix pull request #89: [HELIX-655] Helix per-participant concurrent task th...

2017-05-15 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/89#discussion_r116571005 --- Diff: helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java --- @@ -90,60 +96,86 @@ private

[GitHub] helix pull request #89: [HELIX-655] Helix per-participant concurrent task th...

2017-05-15 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/89#discussion_r116559458 --- Diff: helix-core/src/main/java/org/apache/helix/task/JobRebalancer.java --- @@ -704,4 +712,30 @@ private PartitionAssignment(String instance, String

[GitHub] helix pull request #89: [HELIX-655] Helix per-participant concurrent task th...

2017-05-15 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/89#discussion_r116557426 --- Diff: helix-core/src/main/java/org/apache/helix/task/JobRebalancer.java --- @@ -57,6 +57,7 @@ new FixedTargetTaskAssignmentCalculator

[GitHub] helix pull request #89: [HELIX-655] Helix per-participant concurrent task th...

2017-05-15 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/89#discussion_r116556982 --- Diff: helix-core/src/main/java/org/apache/helix/controller/stages/CurrentStateOutput.java --- @@ -260,13 +260,72 @@ public Message getPendingState

[GitHub] helix pull request #89: [HELIX-655] Helix per-participant concurrent task th...

2017-05-15 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/89#discussion_r116555244 --- Diff: helix-core/src/main/java/org/apache/helix/controller/stages/CurrentStateOutput.java --- @@ -260,13 +260,72 @@ public Message getPendingState

[GitHub] helix pull request #89: [HELIX-655] Helix per-participant concurrent task th...

2017-05-15 Thread jiajunwang
Github user jiajunwang commented on a diff in the pull request: https://github.com/apache/helix/pull/89#discussion_r116554664 --- Diff: helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java --- @@ -204,4 +236,28 @@ private MappingCalculator

[GitHub] helix pull request #89: [HELIX-655] Helix per-participant concurrent task th...

2017-05-03 Thread jiajunwang
GitHub user jiajunwang opened a pull request: https://github.com/apache/helix/pull/89 [HELIX-655] Helix per-participant concurrent task throttling Add per participant concurrent task throttling. 1. Add a participant configuration item "MAX_CONCURRENT_TASK" for