[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392232#comment-16392232
 ] 

ASF GitHub Bot commented on HELIX-674:
--

Github user jiajunwang commented on the issue:

https://github.com/apache/helix/pull/137
  
Changed to another branch instead of using master.
https://github.com/apache/helix/pull/145


> Constraint Based Resource Rebalancer
> 
>
> Key: HELIX-674
> URL: https://issues.apache.org/jira/browse/HELIX-674
> Project: Apache Helix
>  Issue Type: New Feature
>Reporter: Jiajun Wang
>Assignee: Jiajun Wang
>Priority: Major
> Fix For: 0.8.x
>
> Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf
>
>
> Helix rebalancer assigns resources according to different strategies. 
> Recently, we optimize the strategy for evenness and minimize movement. 
> However, the evenness here only applies to partition numbers. Moreover, we've 
> got more requests for customizable rebalancer from our users.
> Take partition weight as an example:
> In reality, partition replicas have different size. We use "partition weight" 
> as an abstraction of the partition size. It can be network traffic usage, 
> disk usage, or any other combined factors.
> Given each partition may have different weights, Helix should be able to 
> assign partition accordingly. So that the distribution would be even 
> regarding the weight.
> In this project, we are planning new rebalancer mechanism that generates 
> resource partition assignment according to a list of "constraints". Current 
> rebalance strategy can be regarded as one kind of constraint. Moving forward, 
> Helix users would be able to extend the constraint interface using their own 
> logic.
> Some init discussions are in progress and we will have a proposal posted here 
> soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392231#comment-16392231
 ] 

ASF GitHub Bot commented on HELIX-674:
--

GitHub user jiajunwang opened a pull request:

https://github.com/apache/helix/pull/145

[HELIX-674] Introducing constraints based rebalancing mechanism.

Constraint can be customized by application to restrict how rebalancing is 
processed.
The change also includes examples to demonstrate how constraints can be 
defined and used.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jiajunwang/helix constraint

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/145.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #145


commit 195ca9818617e3909c8bf285912778b711edd67c
Author: jiajunwang 
Date:   2018-02-27T01:21:48Z

[HELIX-674] Introducing constraints based rebalancing mechanism.

Constraint can be customized by application to restrict how rebalancing is 
processed.
The change also includes examples to demonstrate how constraints can be 
defined and used.




> Constraint Based Resource Rebalancer
> 
>
> Key: HELIX-674
> URL: https://issues.apache.org/jira/browse/HELIX-674
> Project: Apache Helix
>  Issue Type: New Feature
>Reporter: Jiajun Wang
>Assignee: Jiajun Wang
>Priority: Major
> Fix For: 0.8.x
>
> Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf
>
>
> Helix rebalancer assigns resources according to different strategies. 
> Recently, we optimize the strategy for evenness and minimize movement. 
> However, the evenness here only applies to partition numbers. Moreover, we've 
> got more requests for customizable rebalancer from our users.
> Take partition weight as an example:
> In reality, partition replicas have different size. We use "partition weight" 
> as an abstraction of the partition size. It can be network traffic usage, 
> disk usage, or any other combined factors.
> Given each partition may have different weights, Helix should be able to 
> assign partition accordingly. So that the distribution would be even 
> regarding the weight.
> In this project, we are planning new rebalancer mechanism that generates 
> resource partition assignment according to a list of "constraints". Current 
> rebalance strategy can be regarded as one kind of constraint. Moving forward, 
> Helix users would be able to extend the constraint interface using their own 
> logic.
> Some init discussions are in progress and we will have a proposal posted here 
> soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392233#comment-16392233
 ] 

ASF GitHub Bot commented on HELIX-674:
--

Github user jiajunwang closed the pull request at:

https://github.com/apache/helix/pull/137


> Constraint Based Resource Rebalancer
> 
>
> Key: HELIX-674
> URL: https://issues.apache.org/jira/browse/HELIX-674
> Project: Apache Helix
>  Issue Type: New Feature
>Reporter: Jiajun Wang
>Assignee: Jiajun Wang
>Priority: Major
> Fix For: 0.8.x
>
> Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf
>
>
> Helix rebalancer assigns resources according to different strategies. 
> Recently, we optimize the strategy for evenness and minimize movement. 
> However, the evenness here only applies to partition numbers. Moreover, we've 
> got more requests for customizable rebalancer from our users.
> Take partition weight as an example:
> In reality, partition replicas have different size. We use "partition weight" 
> as an abstraction of the partition size. It can be network traffic usage, 
> disk usage, or any other combined factors.
> Given each partition may have different weights, Helix should be able to 
> assign partition accordingly. So that the distribution would be even 
> regarding the weight.
> In this project, we are planning new rebalancer mechanism that generates 
> resource partition assignment according to a list of "constraints". Current 
> rebalance strategy can be regarded as one kind of constraint. Moving forward, 
> Helix users would be able to extend the constraint interface using their own 
> logic.
> Some init discussions are in progress and we will have a proposal posted here 
> soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] helix issue #137: [HELIX-674] Introducing constraints based rebalancing mech...

[jiajunwang]

Github user jiajunwang commented on the issue:

https://github.com/apache/helix/pull/137
  
Changed to another branch instead of using master.
https://github.com/apache/helix/pull/145


---

[GitHub] helix pull request #137: [HELIX-674] Introducing constraints based rebalanci...

[jiajunwang]

Github user jiajunwang closed the pull request at:

https://github.com/apache/helix/pull/137


---

[GitHub] helix pull request #145: [HELIX-674] Introducing constraints based rebalanci...

[jiajunwang]

GitHub user jiajunwang opened a pull request:

https://github.com/apache/helix/pull/145

[HELIX-674] Introducing constraints based rebalancing mechanism.

Constraint can be customized by application to restrict how rebalancing is 
processed.
The change also includes examples to demonstrate how constraints can be 
defined and used.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jiajunwang/helix constraint

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/145.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #145


commit 195ca9818617e3909c8bf285912778b711edd67c
Author: jiajunwang 
Date:   2018-02-27T01:21:48Z

[HELIX-674] Introducing constraints based rebalancing mechanism.

Constraint can be customized by application to restrict how rebalancing is 
processed.
The change also includes examples to demonstrate how constraints can be 
defined and used.




---

[GitHub] helix pull request #144: [HELIX-677] Change error log to info/warning when r...

[jiajunwang]

GitHub user jiajunwang opened a pull request:

https://github.com/apache/helix/pull/144

[HELIX-677] Change error log to info/warning when replica count canno…

…t be read in ResourceMonitor.

Some string such as ALL_LIVEINSTANCES in the replica field cannot be 
processed by the monitor. And those resources are not expected to be monitoring.
Currently error log on this case drags too much attention. Lower the log 
level to info/warning according to the Exception type.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jiajunwang/helix helix-677

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/144.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #144


commit 7bdae1f5287b730ee537c972e1a5b77c2f4048e3
Author: Jiajun Wang 
Date:   2018-02-21T22:16:30Z

[HELIX-677] Change error log to info/warning when replica count cannot be 
read in ResourceMonitor.

Some string such as ALL_LIVEINSTANCES in the replica field cannot be 
processed by the monitor. And those resources are not expected to be monitoring.
Currently error log on this case drags too much attention. Lower the log 
level to info/warning according to the Exception type.




---

[jira] [Commented] (HELIX-677) Change error log to info/warning when replica count cannot be read in ResourceMonitor.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392226#comment-16392226
 ] 

ASF GitHub Bot commented on HELIX-677:
--

GitHub user jiajunwang opened a pull request:

https://github.com/apache/helix/pull/144

[HELIX-677] Change error log to info/warning when replica count canno…

…t be read in ResourceMonitor.

Some string such as ALL_LIVEINSTANCES in the replica field cannot be 
processed by the monitor. And those resources are not expected to be monitoring.
Currently error log on this case drags too much attention. Lower the log 
level to info/warning according to the Exception type.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jiajunwang/helix helix-677

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/144.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #144


commit 7bdae1f5287b730ee537c972e1a5b77c2f4048e3
Author: Jiajun Wang 
Date:   2018-02-21T22:16:30Z

[HELIX-677] Change error log to info/warning when replica count cannot be 
read in ResourceMonitor.

Some string such as ALL_LIVEINSTANCES in the replica field cannot be 
processed by the monitor. And those resources are not expected to be monitoring.
Currently error log on this case drags too much attention. Lower the log 
level to info/warning according to the Exception type.




>  Change error log to info/warning when replica count cannot be read in 
> ResourceMonitor.
> ---
>
> Key: HELIX-677
> URL: https://issues.apache.org/jira/browse/HELIX-677
> Project: Apache Helix
>  Issue Type: Task
>Reporter: Jiajun Wang
>Assignee: Jiajun Wang
>Priority: Major
>
> Currently error log on this case drags too much attention. Lower the log 
> level to info/warning according to the Exception type.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HELIX-676) Controller keeps updating idealstates when there is no real diff.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392216#comment-16392216
 ] 

ASF GitHub Bot commented on HELIX-676:
--

GitHub user jiajunwang opened a pull request:

https://github.com/apache/helix/pull/143

[HELIX-676] Fix the issue that the controller keep updating idealstates 
when there is no real diff.

The causes of the problem are that:

1.A previous issue introduced into IntermediateStateCalcStage prevents 
ERROR/OFFLINE state replicas from being added to the intermediateState, given 
the controller thinks recovery rebalance is not necessary.
This makes the processed stateMapping in pipeline always different from the 
cached IdeaStates. And then causes endless updating.

2.Another separate change in persistAssignmentStage is also related to this 
issue. When updating the map/list, we used putAll. This call will keep all 
existing items but only modify the intersect. Our assumption previously is 
allow customized items. However, when we investigate this issue, it would be 
error-prone to allow these customized items in the map/list fields. Helix won't 
be able to tell if one item is added by the controller or users. So we decide 
to use clear and set instead. This ensure the map/list fields are always 
uptodate.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jiajunwang/helix helix-676

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/143.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #143


commit 9083950d9a0e5f5c377672fb5b21af13cdde0d9a
Author: Jiajun Wang 
Date:   2018-02-12T19:45:18Z

[HELIX-676] Fix the issue that the controller keep updating idealstates 
when there is no real diff.

The causes of the problem are that:

1.A previous issue introduced into IntermediateStateCalcStage prevents 
ERROR/OFFLINE state replicas from being added to the intermediateState, given 
the controller thinks recovery rebalance is not necessary.
This makes the processed stateMapping in pipeline always different from the 
cached IdeaStates. And then causes endless updating.

2.Another separate change in persistAssignmentStage is also related to this 
issue. When updating the map/list, we used putAll. This call will keep all 
existing items but only modify the intersect. Our assumption previously is 
allow customized items. However, when we investigate this issue, it would be 
error-prone to allow these customized items in the map/list fields. Helix won't 
be able to tell if one item is added by the controller or users. So we decide 
to use clear and set instead. This ensure the map/list fields are always 
uptodate.

commit 08ff9d0b287ece2a17f2430ba82bd4f2373263e7
Author: Jiajun Wang 
Date:   2018-02-17T00:31:27Z

[HELIX-676] Adding test for intermediate state cal stage.

Testing 4 cases (recovery, load balance, recovery with transient states, 
and error partition blocks load balance).
This test covers the recent fix for HELIX-676.




> Controller keeps updating idealstates when there is no real diff.
> -
>
> Key: HELIX-676
> URL: https://issues.apache.org/jira/browse/HELIX-676
> Project: Apache Helix
>  Issue Type: Bug
>Reporter: Jiajun Wang
>Assignee: Jiajun Wang
>Priority: Major
>
> An issue has been confirmed that controller may keep updating ideastates when 
> PERSIST__STATE is true.
> This increase ZK traffic a lot, and cause performance issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] helix pull request #143: [HELIX-676] Fix the issue that the controller keep ...

[jiajunwang]

GitHub user jiajunwang opened a pull request:

https://github.com/apache/helix/pull/143

[HELIX-676] Fix the issue that the controller keep updating idealstates 
when there is no real diff.

The causes of the problem are that:

1.A previous issue introduced into IntermediateStateCalcStage prevents 
ERROR/OFFLINE state replicas from being added to the intermediateState, given 
the controller thinks recovery rebalance is not necessary.
This makes the processed stateMapping in pipeline always different from the 
cached IdeaStates. And then causes endless updating.

2.Another separate change in persistAssignmentStage is also related to this 
issue. When updating the map/list, we used putAll. This call will keep all 
existing items but only modify the intersect. Our assumption previously is 
allow customized items. However, when we investigate this issue, it would be 
error-prone to allow these customized items in the map/list fields. Helix won't 
be able to tell if one item is added by the controller or users. So we decide 
to use clear and set instead. This ensure the map/list fields are always 
uptodate.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jiajunwang/helix helix-676

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/143.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #143


commit 9083950d9a0e5f5c377672fb5b21af13cdde0d9a
Author: Jiajun Wang 
Date:   2018-02-12T19:45:18Z

[HELIX-676] Fix the issue that the controller keep updating idealstates 
when there is no real diff.

The causes of the problem are that:

1.A previous issue introduced into IntermediateStateCalcStage prevents 
ERROR/OFFLINE state replicas from being added to the intermediateState, given 
the controller thinks recovery rebalance is not necessary.
This makes the processed stateMapping in pipeline always different from the 
cached IdeaStates. And then causes endless updating.

2.Another separate change in persistAssignmentStage is also related to this 
issue. When updating the map/list, we used putAll. This call will keep all 
existing items but only modify the intersect. Our assumption previously is 
allow customized items. However, when we investigate this issue, it would be 
error-prone to allow these customized items in the map/list fields. Helix won't 
be able to tell if one item is added by the controller or users. So we decide 
to use clear and set instead. This ensure the map/list fields are always 
uptodate.

commit 08ff9d0b287ece2a17f2430ba82bd4f2373263e7
Author: Jiajun Wang 
Date:   2018-02-17T00:31:27Z

[HELIX-676] Adding test for intermediate state cal stage.

Testing 4 cases (recovery, load balance, recovery with transient states, 
and error partition blocks load balance).
This test covers the recent fix for HELIX-676.




---

[jira] [Commented] (HELIX-675) Cluster Monitor may not be closed or init correctly when leadership changes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392193#comment-16392193
 ] 

ASF GitHub Bot commented on HELIX-675:
--

GitHub user jiajunwang opened a pull request:

https://github.com/apache/helix/pull/142

[HELIX-675] Refactor controller start/cleanup logic to ensure monitor…

… register/reset is handled in any event orders.

Due to different possible event process order, controller init event might 
be processed later or earlier than expected.
This cause inconsistency when even handler thread process and record 
information in the cluster monitor.
This change ensures cluster monitor is off when the leadership changes to 
other node. So no extra metric data will be generated.

Also upgrade related test cases to verify MBean counts.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jiajunwang/helix helix-675

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/142.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #142


commit f5017c35d2971bb8a5de7b8defca77c0ffdd13bb
Author: jiajunwang 
Date:   2018-03-09T00:51:37Z

[HELIX-675] Refactor controller start/cleanup logic to ensure monitor 
register/reset is handled in any event orders.

Due to different possible event process order, controller init event might 
be processed later or earlier than expected.
This cause inconsistency when even handler thread process and record 
information in the cluster monitor.
This change ensures cluster monitor is off when the leadership changes to 
other node. So no extra metric data will be generated.

Also upgrade related test cases to verify MBean counts.




> Cluster Monitor may not be closed or init correctly when leadership changes
> ---
>
> Key: HELIX-675
> URL: https://issues.apache.org/jira/browse/HELIX-675
> Project: Apache Helix
>  Issue Type: Task
>Reporter: Jiajun Wang
>Assignee: Jiajun Wang
>Priority: Major
>
> Due to different possible event process order, controller init event might be 
> processed later or earlier than expected.
> This cause inconsistency when even handler thread process and record 
> information in the cluster monitor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] helix pull request #142: [HELIX-675] Refactor controller start/cleanup logic...

[jiajunwang]

GitHub user jiajunwang opened a pull request:

https://github.com/apache/helix/pull/142

[HELIX-675] Refactor controller start/cleanup logic to ensure monitor…

… register/reset is handled in any event orders.

Due to different possible event process order, controller init event might 
be processed later or earlier than expected.
This cause inconsistency when even handler thread process and record 
information in the cluster monitor.
This change ensures cluster monitor is off when the leadership changes to 
other node. So no extra metric data will be generated.

Also upgrade related test cases to verify MBean counts.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jiajunwang/helix helix-675

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/142.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #142


commit f5017c35d2971bb8a5de7b8defca77c0ffdd13bb
Author: jiajunwang 
Date:   2018-03-09T00:51:37Z

[HELIX-675] Refactor controller start/cleanup logic to ensure monitor 
register/reset is handled in any event orders.

Due to different possible event process order, controller init event might 
be processed later or earlier than expected.
This cause inconsistency when even handler thread process and record 
information in the cluster monitor.
This change ensures cluster monitor is off when the leadership changes to 
other node. So no extra metric data will be generated.

Also upgrade related test cases to verify MBean counts.




---

helix - Build # 1409 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built helix (build #1409)

Status: Still Failing

Check console output at https://builds.apache.org/job/helix/1409/ to view the 
results.

[GitHub] helix pull request #136: [helix-core] _rebalanceTimerPeriod in RebalanceConf...

[juvenzhu]

Github user juvenzhu closed the pull request at:

https://github.com/apache/helix/pull/136


---

[GitHub] helix pull request #141: Fix the job parents listing logic in REST & Make lo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/helix/pull/141


---

[GitHub] helix pull request #141: Fix the job parents listing logic in REST & Make lo...

[lei-xia]

GitHub user lei-xia opened a pull request:

https://github.com/apache/helix/pull/141

 Fix the job parents listing logic in REST  &  Make logs are not printed 
with empty list in intermediate stage 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lei-xia/helix master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/141.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #141


commit 5c5186bf6f24720875710efe99fd389d4dc91602
Author: Junkai Xue 
Date:   2018-02-02T00:19:21Z

Fix the job parents listing logic in REST

commit 5f142f2ec7dcf718111bbc9e94eb83d7689428f0
Author: Junkai Xue 
Date:   2018-02-02T21:52:50Z

Make logs are not printed with empty list in intermediate stage




---

[GitHub] helix issue #136: [helix-core] _rebalanceTimerPeriod in RebalanceConfig shou...

[lei-xia]

Github user lei-xia commented on the issue:

https://github.com/apache/helix/pull/136
  
@juvenzhu  already merged, somehow github did not close this PR, please go 
ahead to close it.  Thanks!


---

[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

[Jiajun Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392019#comment-16392019
 ] 

Jiajun Wang commented on HELIX-674:
---

Due to lack of Apache Wiki privilege, I cannot create a wiki page for posting 
design doc there.

Upload the design doc as PDF for now.

> Constraint Based Resource Rebalancer
> 
>
> Key: HELIX-674
> URL: https://issues.apache.org/jira/browse/HELIX-674
> Project: Apache Helix
>  Issue Type: New Feature
>Reporter: Jiajun Wang
>Assignee: Jiajun Wang
>Priority: Major
> Fix For: 0.8.x
>
> Attachments: Constraint-BasedResourceRebalancing-080318-2226-240.pdf
>
>
> Helix rebalancer assigns resources according to different strategies. 
> Recently, we optimize the strategy for evenness and minimize movement. 
> However, the evenness here only applies to partition numbers. Moreover, we've 
> got more requests for customizable rebalancer from our users.
> Take partition weight as an example:
> In reality, partition replicas have different size. We use "partition weight" 
> as an abstraction of the partition size. It can be network traffic usage, 
> disk usage, or any other combined factors.
> Given each partition may have different weights, Helix should be able to 
> assign partition accordingly. So that the distribution would be even 
> regarding the weight.
> In this project, we are planning new rebalancer mechanism that generates 
> resource partition assignment according to a list of "constraints". Current 
> rebalance strategy can be regarded as one kind of constraint. Moving forward, 
> Helix users would be able to extend the constraint interface using their own 
> logic.
> Some init discussions are in progress and we will have a proposal posted here 
> soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

helix - Build # 1408 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built helix (build #1408)

Status: Still Failing

Check console output at https://builds.apache.org/job/helix/1408/ to view the 
results.

[GitHub] helix pull request #140: [HELIX-679] consolidate semantics of recursively de...

[zhan849]

GitHub user zhan849 opened a pull request:

https://github.com/apache/helix/pull/140

[HELIX-679] consolidate semantics of recursively delete path in ZkClient

This change consolidates semantics of APIs in ZkClient that recursively 
deletes a path

* For backward compatibility, we keep `deleteRecursive()`, which will only 
return true/false, and will not throw exception.
* create a new method called deleteRecursively() that will only throw 
exception upon error.
* mark `deleteRecursive()` as deprecated as throwing exception can carry 
error information
* make all current usage of `deleteRecursive()` to `deleteRecursively()` 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhan849/helix harry/zk-client-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/140.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #140


commit 8412d3f7e7d8097eee820c4d055b1526ac74aca1
Author: hrzhang 
Date:   2018-03-08T22:04:42Z

[HELIX-679] consolidate semantics of recursively delete path in ZkClient




---

[jira] [Commented] (HELIX-679) Consolidated behaviors for deleteRecursive and deleteRecursively in ZkClient

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391988#comment-16391988
 ] 

ASF GitHub Bot commented on HELIX-679:
--

GitHub user zhan849 opened a pull request:

https://github.com/apache/helix/pull/140

[HELIX-679] consolidate semantics of recursively delete path in ZkClient

This change consolidates semantics of APIs in ZkClient that recursively 
deletes a path

* For backward compatibility, we keep `deleteRecursive()`, which will only 
return true/false, and will not throw exception.
* create a new method called deleteRecursively() that will only throw 
exception upon error.
* mark `deleteRecursive()` as deprecated as throwing exception can carry 
error information
* make all current usage of `deleteRecursive()` to `deleteRecursively()` 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhan849/helix harry/zk-client-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/140.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #140


commit 8412d3f7e7d8097eee820c4d055b1526ac74aca1
Author: hrzhang 
Date:   2018-03-08T22:04:42Z

[HELIX-679] consolidate semantics of recursively delete path in ZkClient




> Consolidated behaviors for deleteRecursive and deleteRecursively in ZkClient
> 
>
> Key: HELIX-679
> URL: https://issues.apache.org/jira/browse/HELIX-679
> Project: Apache Helix
>  Issue Type: Bug
>  Components: helix-core
>Reporter: Hao Zhang
>Priority: Major
>
> According to it's documentation `deleteRecursive()` should return true if 
> operation is successful else false. But the semantics of the base function 
> (`delete()`) it calls is different: it returns true if operation is 
> successful, returns false if node does not exist, throws exception upon other 
> errors, and therefore `deleteRecursive()` will also throw exception, and will 
> return false if any sub-path is deleted already, which is confusing
> To consolidate semantics, we should either have the function only return 
> true/false or only throw exception upon error.
>  
> Also, to make change backward compatible, I'd propose the following change:
>  # deleteRecursive() will only return true/false, and will not throw 
> exception. If subpath does not exist, it should consider successful
>  # create a new method called deleteRecursively() that will only throw 
> exception upon error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

helix - Build # 1407 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built helix (build #1407)

Status: Still Failing

Check console output at https://builds.apache.org/job/helix/1407/ to view the 
results.

[jira] [Created] (HELIX-679) Consolidated behaviors for deleteRecursive and deleteRecursively in ZkClient

[Hao Zhang (JIRA)]

Hao Zhang created HELIX-679:
---

 Summary: Consolidated behaviors for deleteRecursive and 
deleteRecursively in ZkClient
 Key: HELIX-679
 URL: https://issues.apache.org/jira/browse/HELIX-679
 Project: Apache Helix
  Issue Type: Bug
  Components: helix-core
Reporter: Hao Zhang


According to it's documentation `deleteRecursive()` should return true if 
operation is successful else false. But the semantics of the base function 
(`delete()`) it calls is different: it returns true if operation is successful, 
returns false if node does not exist, throws exception upon other errors, and 
therefore `deleteRecursive()` will also throw exception, and will return false 
if any sub-path is deleted already, which is confusing

To consolidate semantics, we should either have the function only return 
true/false or only throw exception upon error.

 

Also, to make change backward compatible, I'd propose the following change:
 # deleteRecursive() will only return true/false, and will not throw exception. 
If subpath does not exist, it should consider successful
 # create a new method called deleteRecursively() that will only throw 
exception upon error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

helix - Build # 1406 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built helix (build #1406)

Status: Still Failing

Check console output at https://builds.apache.org/job/helix/1406/ to view the 
results.

[jira] [Created] (HELIX-678) Clear cluster even queue when controller is no longer leader

[Jiajun Wang (JIRA)]

Jiajun Wang created HELIX-678:
-

 Summary: Clear cluster even queue when controller is no longer 
leader
 Key: HELIX-678
 URL: https://issues.apache.org/jira/browse/HELIX-678
 Project: Apache Helix
  Issue Type: Task
Reporter: Jiajun Wang
Assignee: Jiajun Wang


Currently, the controller will keep processing queued event even it is no 
longer the leader.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HELIX-677) Change error log to info/warning when replica count cannot be read in ResourceMonitor.

[Jiajun Wang (JIRA)]

Jiajun Wang created HELIX-677:
-

 Summary:  Change error log to info/warning when replica count 
cannot be read in ResourceMonitor.
 Key: HELIX-677
 URL: https://issues.apache.org/jira/browse/HELIX-677
 Project: Apache Helix
  Issue Type: Task
Reporter: Jiajun Wang
Assignee: Jiajun Wang


Currently error log on this case drags too much attention. Lower the log level 
to info/warning according to the Exception type.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] helix pull request #139: Two small changes

[lei-xia]

GitHub user lei-xia opened a pull request:

https://github.com/apache/helix/pull/139

Two small changes

- Remove duplicated log lines
- Fix issue in reporting MissingMinActiveReplicaPartitionGauge metric in 
ResourceMonitor when there is no IdealMapping persisted in IdealState.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lei-xia/helix master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/helix/pull/139.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #139


commit d9eb0e5ff9fe859e2ea16028838c4e82ccdf5da0
Author: Lei Xia 
Date:   2018-01-25T19:52:45Z

Remove duplicated log lines.

commit f31d6465944adb4e4a6b66f4e35538610e6f4bbd
Author: Lei Xia 
Date:   2018-01-30T02:12:56Z

Fix issue in reporting MissingMinActiveReplicaPartitionGauge metric in 
ResourceMonitor when there is no IdealMapping persisted in IdealState.




---

[jira] [Created] (HELIX-676) Controller keeps updating idealstates when there is no real diff.

[Jiajun Wang (JIRA)]

Jiajun Wang created HELIX-676:
-

 Summary: Controller keeps updating idealstates when there is no 
real diff.
 Key: HELIX-676
 URL: https://issues.apache.org/jira/browse/HELIX-676
 Project: Apache Helix
  Issue Type: Bug
Reporter: Jiajun Wang
Assignee: Jiajun Wang


An issue has been confirmed that controller may keep updating ideastates when 
PERSIST__STATE is true.

This increase ZK traffic a lot, and cause performance issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HELIX-675) Cluster Monitor may not be closed or init correctly when leadership changes

[Jiajun Wang (JIRA)]

Jiajun Wang created HELIX-675:
-

 Summary: Cluster Monitor may not be closed or init correctly when 
leadership changes
 Key: HELIX-675
 URL: https://issues.apache.org/jira/browse/HELIX-675
 Project: Apache Helix
  Issue Type: Task
Reporter: Jiajun Wang
Assignee: Jiajun Wang


Due to different possible event process order, controller init event might be 
processed later or earlier than expected.
This cause inconsistency when even handler thread process and record 
information in the cluster monitor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HELIX-674) Constraint Based Resource Rebalancer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HELIX-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391771#comment-16391771
 ] 

ASF GitHub Bot commented on HELIX-674:
--

Github user jiajunwang commented on the issue:

https://github.com/apache/helix/pull/137
  
@guoyuepeng Thanks for the comment. Fixed.

Also updated all TODOs, and refactored the demo features with test cases.


> Constraint Based Resource Rebalancer
> 
>
> Key: HELIX-674
> URL: https://issues.apache.org/jira/browse/HELIX-674
> Project: Apache Helix
>  Issue Type: New Feature
>Reporter: Jiajun Wang
>Assignee: Jiajun Wang
>Priority: Major
> Fix For: 0.8.x
>
>
> Helix rebalancer assigns resources according to different strategies. 
> Recently, we optimize the strategy for evenness and minimize movement. 
> However, the evenness here only applies to partition numbers. Moreover, we've 
> got more requests for customizable rebalancer from our users.
> Take partition weight as an example:
> In reality, partition replicas have different size. We use "partition weight" 
> as an abstraction of the partition size. It can be network traffic usage, 
> disk usage, or any other combined factors.
> Given each partition may have different weights, Helix should be able to 
> assign partition accordingly. So that the distribution would be even 
> regarding the weight.
> In this project, we are planning new rebalancer mechanism that generates 
> resource partition assignment according to a list of "constraints". Current 
> rebalance strategy can be regarded as one kind of constraint. Moving forward, 
> Helix users would be able to extend the constraint interface using their own 
> logic.
> Some init discussions are in progress and we will have a proposal posted here 
> soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] helix issue #137: [HELIX-674] Introducing constraints based rebalancing mech...

[jiajunwang]

Github user jiajunwang commented on the issue:

https://github.com/apache/helix/pull/137
  
@guoyuepeng Thanks for the comment. Fixed.

Also updated all TODOs, and refactored the demo features with test cases.


---

[GitHub] helix pull request #138: CallbackHandler to use either java config property ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/helix/pull/138


---