This is an automated email from the ASF dual-hosted git repository.
jiajunwang pushed a change to branch wagedRebalancer
in repository https://gitbox.apache.org/repos/asf/helix.git.
discard 302e1e7 Asynchronously calculating the Baseline (#632)
omit 4206500 Refine methods name and comments. (#664)
omit 138d1c6 Refine the WAGED rebalancer to minimize the partial rebalance
workload. (#639)
omit f6cd078 Improve the algorithm so it prioritizes the assignment to the
idle nodes when the constraint evaluation results are the same (#651)
omit 95a7339 Decouple the event type and the scheduled rebalance cache
refresh option. (#638)
omit 17b5c93 Add new WAGED rebalancer config item
"GLOBAL_REBALANCE_ASYNC_MODE". (#637)
omit 7bb4f2b Simply and enhance the RebalanceLatencyGauge so it can be
used in multi-threads. (#636)
omit 6aa8184 Fix a potential issue in the ResourceChangeSnapshot. (#635)
omit 03c0b30 Add REST API endpoints for WAGED Rebalancer (#611)
omit ac0256e Fix the unstable test TestZeroReplicaAvoidance. (#603)
omit 5f51ff4 Improve the WAGED rebalancer performance. (#586)
omit 0d7033a Change calculation for baseline divergence. (#598)
omit ca93db4 Add Java API for adding and validating resources for WAGED
rebalancer (#570)
omit 524c258 Change ClusterConfig.setDefaultCapacityMap to be private.
(#590)
omit be1a4f6 Introduce Dry-run Waged Rebalancer for the verifiers and
tests. (#573)
omit 6b02162 Filter resource map with ideal states for instance capacity
metrics. (#574)
omit 5fcf0f8 Avoid redundant writes in AssignmentMetadataStore (#564)
omit 19fa29c Fix formula incorrection in the comment for measuring
baseline divergence. (#559)
omit 9bc8060 Add max capacity usage metric for instance monitor. (#548)
omit 153a59b Use a deep copy of the new best possible assignment for
measuring baseline divergence. (#542)
omit 1631bae Refine ivy file config. The org were not configured
correctly. (#551)
omit 4d4418b Add the new Rebalancer monitor domain to the active domain
list. (#550)
omit ee212de Fix a missing parameter when the WAGED rebalancer init the
change detector. (#547)
omit 11fe5f1 Modify the ivy file to add the new math3 lib dependency.
(#546)
omit e002a31 Implement increment() method in CountMetric class. (#537)
omit 9f77982 Adjust the replica rebalance calculating ordering to avoid
static order. (#535)
omit 9af4c4f Minor fix for the constraints related tests. (#545)
omit f17fcdc Refactor soft constraints to simply the algorithm and fix
potential issues. (#520)
omit 788ffcf Change change detector to a regular field in the WAGED
rebalancer instead of static threadlocal. (#543)
omit cdc7109 Make WagedRebalancer static by creating a ThreadLocal (#540)
omit 1842990 Refine the rebalance scope calculating logic in the WAGED
rebalancer. (#519)
omit 02141ab Implement monitoring mbeans for the WAGED rebalancer. (#525)
omit 21bf399 Make log clearer after finishing calculateAssignment. (#531)
omit eb78edc The WAGED rebalancer returns the previously calculated
assignment on calculation failure (#514)
omit 8f9b3e2 Increase parallelism for ZkBucketDataAccessor (#506)
omit def2856 More strict partition weight validation while creating the
cluster model. (#511)
omit 0e698d8 Fixing rebalance cache issue and stablize the tests. (#510)
omit dadc021 Add latency metric components for WAGED rebalancer (#490)
omit 3f1c294 Load soft constraint weight from resources/properties file
(#492)
omit d797b4c Adjust the topology processing logic for instance to ensure
backward compatibility.
omit 98778ed Enable maintenance mode for the WAGED rebalancer.
omit 4ad6056 Add delayed rebalance and user-defined preference list
features to the WAGED rebalancer. (#456)
omit 06c9f96 Separate AssignableNode properties by Immutable and Mutable
(#485)
omit 64ae736 Integrate the WAGED rebalancer with all the related
components. (#466)
omit 95a2e4e Add the remaining implementation of
ConstraintBasedAlgorithmFactory (#478)
omit fbfb893 PartitionMovementSoftConstraint Implementation (#474)
omit 77ecdb9 Adjust the expected replica count according to fault zone
count. (#476)
omit ca8a114 Improve ResourceTopStateAntiAffinityConstraint (#475)
omit e858448 Add soft constraint: ResourcePartitionAntiAffinityConstraint
(#464)
omit 6b29b72 Implement MaxCapacityUsageInstanceConstraint soft constraint
(#463)
omit c7a303b Add soft constraint: ResourcetopStateAntiAffinityConstraint
(#465)
omit 313d23b Implement one of the soft constraints (#450)
omit 5d8dbcd Fix TestWagedRebalancer and add constructor in
AssignmentMetadataStore
omit baaabf2 Implement AssignmentMetadataStore (#453)
omit ab5745c HardConstraints Implementation and unit tests (#433)
omit 723331d Implement the WAGED rebalancer with the limited
functionality. (#443)
omit ec60cce Validate the instance capacity/partition weight configuration
while constructing the assignable instances (#451)
omit dc59a6e Implement the basic constraint based algorithm (#381)
omit a1c4b81 Add BucketDataAccessor for large writes
omit 2300d2b Record the replica objects in the AssignableNode in addition
to the partition name (#440)
omit 8e599f8 Add special treatment for ClusterConfig
omit bef2f47 Modify the expected change type from CONFIG to CLUSTER_CONFIG
in the WAGED rebalancer. (#438)
omit 675eadb Resubmit the change: Refine the WAGED rebalancer related
interfaces for integration (#431)
omit 1a65289 Revert "Refine the WAGED rebalancer related interfaces for
integration (#431)" (#437)
omit b63dcde Refine the WAGED rebalancer related interfaces for
integration (#431)
omit dd3641c Redefine the hard/soft constraints (#422)
omit f7c249d Add ChangeDetector interface and ResourceChangeDetector
implementation (#388)
omit f9c2b6f Add cluster level default instance config. (#413)
omit b2bd287 Implement Cluster Model Provider. (#392)
omit 3496a04 Convert all the internal assignment state objects to be
ResourceAssignment. (#399)
omit a1ca120 Change the rebalancer assignment record to be
ResourceAssignment instead of IdealState. (#398)
omit f5065be Implement the WAGED rebalancer cluster model (#362)
omit 69296f9 Adding the configuration items of the WAGED rebalancer. (#348)
omit 1652bf0 Define the WAGED rebalancer interfaces.
add 2416922 Add an option for using dedicated ZkConnection in
ZkBaseDataAccessor (#655)
add 660ae7c Add new interface IZkStateListener to provide session aware
handleNewSession for ZkHelixManager (#644)
add 9adb6a2 Patching zkClient leakage fix #656 (#657)
add f82106c [maven-release-plugin] prepare release helix-0.9.2
add 7b852b0 [maven-release-plugin] prepare for next development iteration
add e61b8ec Revert "[maven-release-plugin] prepare for next development
iteration"
add 6193fb0 Revert "[maven-release-plugin] prepare release helix-0.9.2"
add d6ba1f3 Use ZnRecordSerializer in ConfigAccessor (#666)
new a500eb2 Define the WAGED rebalancer interfaces.
new 864d4e1 Adding the configuration items of the WAGED rebalancer. (#348)
new 00597da Implement the WAGED rebalancer cluster model (#362)
new 603d362 Change the rebalancer assignment record to be
ResourceAssignment instead of IdealState. (#398)
new 5371c30 Convert all the internal assignment state objects to be
ResourceAssignment. (#399)
new 867431a Implement Cluster Model Provider. (#392)
new 73e4a5a Add cluster level default instance config. (#413)
new 103513a Add ChangeDetector interface and ResourceChangeDetector
implementation (#388)
new e8ca755 Redefine the hard/soft constraints (#422)
new 23243b7 Refine the WAGED rebalancer related interfaces for
integration (#431)
new 87a099f Revert "Refine the WAGED rebalancer related interfaces for
integration (#431)" (#437)
new de05f40 Resubmit the change: Refine the WAGED rebalancer related
interfaces for integration (#431)
new 3b4043c Modify the expected change type from CONFIG to CLUSTER_CONFIG
in the WAGED rebalancer. (#438)
new 6b16d86 Add special treatment for ClusterConfig
new 0411562 Record the replica objects in the AssignableNode in addition
to the partition name (#440)
new 9ec9739 Add BucketDataAccessor for large writes
new 415c0e8 Implement the basic constraint based algorithm (#381)
new cb3ccdf Validate the instance capacity/partition weight configuration
while constructing the assignable instances (#451)
new 8926f0d Implement the WAGED rebalancer with the limited
functionality. (#443)
new b9b7f7c HardConstraints Implementation and unit tests (#433)
new 06807a8 Implement AssignmentMetadataStore (#453)
new c0e5c7e Fix TestWagedRebalancer and add constructor in
AssignmentMetadataStore
new 75638e2 Implement one of the soft constraints (#450)
new 0026ce1 Add soft constraint: ResourcetopStateAntiAffinityConstraint
(#465)
new 94cc6ec Implement MaxCapacityUsageInstanceConstraint soft constraint
(#463)
new e589218 Add soft constraint: ResourcePartitionAntiAffinityConstraint
(#464)
new 362d182 Improve ResourceTopStateAntiAffinityConstraint (#475)
new f1281bc Adjust the expected replica count according to fault zone
count. (#476)
new 9e22b66 PartitionMovementSoftConstraint Implementation (#474)
new 5b4baf2 Add the remaining implementation of
ConstraintBasedAlgorithmFactory (#478)
new 360915e Integrate the WAGED rebalancer with all the related
components. (#466)
new be49061 Separate AssignableNode properties by Immutable and Mutable
(#485)
new 70e3aba Add delayed rebalance and user-defined preference list
features to the WAGED rebalancer. (#456)
new 8ed9faf Enable maintenance mode for the WAGED rebalancer.
new 0f19de8 Adjust the topology processing logic for instance to ensure
backward compatibility.
new 31c6a9d Load soft constraint weight from resources/properties file
(#492)
new 11ca52b Add latency metric components for WAGED rebalancer (#490)
new 511ff13 Fixing rebalance cache issue and stablize the tests. (#510)
new f1b6e7b More strict partition weight validation while creating the
cluster model. (#511)
new c92e5a2 Increase parallelism for ZkBucketDataAccessor (#506)
new ce9c6f9 The WAGED rebalancer returns the previously calculated
assignment on calculation failure (#514)
new 680cecf Make log clearer after finishing calculateAssignment. (#531)
new 56f3fa3 Implement monitoring mbeans for the WAGED rebalancer. (#525)
new f6f74ea Refine the rebalance scope calculating logic in the WAGED
rebalancer. (#519)
new 1335921 Make WagedRebalancer static by creating a ThreadLocal (#540)
new 14f70ee Change change detector to a regular field in the WAGED
rebalancer instead of static threadlocal. (#543)
new fbd16d4 Refactor soft constraints to simply the algorithm and fix
potential issues. (#520)
new 8fa6e36 Minor fix for the constraints related tests. (#545)
new d1d938a Adjust the replica rebalance calculating ordering to avoid
static order. (#535)
new e6e9ff5 Implement increment() method in CountMetric class. (#537)
new 18601c9 Modify the ivy file to add the new math3 lib dependency.
(#546)
new 2574f07 Fix a missing parameter when the WAGED rebalancer init the
change detector. (#547)
new 330c7e7 Add the new Rebalancer monitor domain to the active domain
list. (#550)
new de301d9 Refine ivy file config. The org were not configured
correctly. (#551)
new 45916ab Use a deep copy of the new best possible assignment for
measuring baseline divergence. (#542)
new 0833fa9 Add max capacity usage metric for instance monitor. (#548)
new af0e513 Fix formula incorrection in the comment for measuring
baseline divergence. (#559)
new 44c9666 Avoid redundant writes in AssignmentMetadataStore (#564)
new 29fcaec Filter resource map with ideal states for instance capacity
metrics. (#574)
new 93e0d8a Introduce Dry-run Waged Rebalancer for the verifiers and
tests. (#573)
new dc23170 Change ClusterConfig.setDefaultCapacityMap to be private.
(#590)
new c5122b3 Add Java API for adding and validating resources for WAGED
rebalancer (#570)
new 1b1e072 Change calculation for baseline divergence. (#598)
new 55457c3 Improve the WAGED rebalancer performance. (#586)
new 5118725 Fix the unstable test TestZeroReplicaAvoidance. (#603)
new e4a6bc8 Add REST API endpoints for WAGED Rebalancer (#611)
new 3948913 Fix a potential issue in the ResourceChangeSnapshot. (#635)
new eb032c5 Simply and enhance the RebalanceLatencyGauge so it can be
used in multi-threads. (#636)
new 345a25c Add new WAGED rebalancer config item
"GLOBAL_REBALANCE_ASYNC_MODE". (#637)
new e7e9c71 Decouple the event type and the scheduled rebalance cache
refresh option. (#638)
new bcb4f28 Improve the algorithm so it prioritizes the assignment to the
idle nodes when the constraint evaluation results are the same (#651)
new 9435d30 Refine the WAGED rebalancer to minimize the partial rebalance
workload. (#639)
new 0a0142b Refine methods name and comments. (#664)
new 8e77c1e Asynchronously calculating the Baseline (#632)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (302e1e7)
\
N -- N -- N refs/heads/wagedRebalancer (8e77c1e)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 74 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../main/java/org/apache/helix/ConfigAccessor.java | 14 +-
.../apache/helix/manager/zk/ZKHelixManager.java | 43 +++++-
.../java/org/apache/helix/manager/zk/ZKUtil.java | 94 +++++++++----
.../helix/manager/zk/ZkBaseDataAccessor.java | 107 ++++++++++++--
.../helix/manager/zk/ZkCacheBaseDataAccessor.java | 29 +++-
.../apache/helix/manager/zk/ZkCallbackCache.java | 24 +++-
.../helix/manager/zk/client/HelixZkClient.java | 105 +++++++++++++-
.../manager/zk/zookeeper/IZkStateListener.java | 61 ++++++++
.../helix/manager/zk/zookeeper/ZkClient.java | 95 ++++++++++++-
.../helix/manager/zk/zookeeper/ZkEventThread.java | 26 +++-
.../helix/store/zk/ZkHelixPropertyStore.java | 5 +
.../java/org/apache/helix/tools/MessagePoster.java | 20 +--
.../apache/helix/tools/commandtools/ZKDumper.java | 32 +++--
.../java/org/apache/helix/TestConfigAccessor.java | 10 ++
.../test/java/org/apache/helix/ZkTestHelper.java | 18 +--
.../java/org/apache/helix/common/ZkTestBase.java | 6 +-
.../helix/manager/zk/TestHandleNewSession.java | 4 +-
.../apache/helix/manager/zk/TestRawZkClient.java | 153 ++++++++++++++++++---
.../helix/manager/zk/TestZkClusterManager.java | 2 +-
.../apache/helix/manager/zk/TestZkFlapping.java | 4 +-
20 files changed, 729 insertions(+), 123 deletions(-)
create mode 100644
helix-core/src/main/java/org/apache/helix/manager/zk/zookeeper/IZkStateListener.java
