[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-27 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416662#comment-16416662
 ] 

Hudson commented on HBASE-20186:


Results for branch HBASE-19064
[build #77 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/77/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/77//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/77//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/77//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when 
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
>  Issue Type: Improvement
>  Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.3
>
> Attachments: HBASE-20186.master.000.patch, 
> HBASE-20186.master.001.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map 
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions 
> which have been already placed according to the rsgroup assignment, and it is 
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite 
> efficient, as it does not make good use of the fact that servers in 
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
>   for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
>   groupClusterState.put(curr, correctedState.get(curr));
> }
>   }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each 
> rsgroup in average, the code above has time complexity as O(m * n), while 
> using TreeSet's contains(), the time complexity could be reduced to O (m * 
> logn).
> Another improvement is we do not need to scan every server for each rsgroup. 
> If the processed server could be recorded,  we could skip those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-17 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403715#comment-16403715
 ] 

Hudson commented on HBASE-20186:


Results for branch branch-1.4
[build #258 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/258/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/258//General_Nightly_Build_Report/]


(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/258//JDK7_Nightly_Build_Report/]


(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/258//JDK8_Nightly_Build_Report_(Hadoop2)/]




(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when 
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
>  Issue Type: Improvement
>  Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.3
>
> Attachments: HBASE-20186.master.000.patch, 
> HBASE-20186.master.001.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map 
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions 
> which have been already placed according to the rsgroup assignment, and it is 
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite 
> efficient, as it does not make good use of the fact that servers in 
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
>   for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
>   groupClusterState.put(curr, correctedState.get(curr));
> }
>   }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each 
> rsgroup in average, the code above has time complexity as O(m * n), while 
> using TreeSet's contains(), the time complexity could be reduced to O (m * 
> logn).
> Another improvement is we do not need to scan every server for each rsgroup. 
> If the processed server could be recorded,  we could skip those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-17 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403369#comment-16403369
 ] 

Hudson commented on HBASE-20186:


Results for branch branch-2
[build #495 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/495/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/495//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/495//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/495//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when 
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
>  Issue Type: Improvement
>  Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.3
>
> Attachments: HBASE-20186.master.000.patch, 
> HBASE-20186.master.001.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map 
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions 
> which have been already placed according to the rsgroup assignment, and it is 
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite 
> efficient, as it does not make good use of the fact that servers in 
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
>   for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
>   groupClusterState.put(curr, correctedState.get(curr));
> }
>   }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each 
> rsgroup in average, the code above has time complexity as O(m * n), while 
> using TreeSet's contains(), the time complexity could be reduced to O (m * 
> logn).
> Another improvement is we do not need to scan every server for each rsgroup. 
> If the processed server could be recorded,  we could skip those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-17 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403366#comment-16403366
 ] 

Hudson commented on HBASE-20186:


Results for branch branch-1
[build #252 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/252/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/252//General_Nightly_Build_Report/]


(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/252//JDK7_Nightly_Build_Report/]


(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/252//JDK8_Nightly_Build_Report_(Hadoop2)/]




(x) {color:red}-1 source release artifact{color}
-- See build output for details.


> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when 
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
>  Issue Type: Improvement
>  Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.3
>
> Attachments: HBASE-20186.master.000.patch, 
> HBASE-20186.master.001.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map 
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions 
> which have been already placed according to the rsgroup assignment, and it is 
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite 
> efficient, as it does not make good use of the fact that servers in 
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
>   for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
>   groupClusterState.put(curr, correctedState.get(curr));
> }
>   }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each 
> rsgroup in average, the code above has time complexity as O(m * n), while 
> using TreeSet's contains(), the time complexity could be reduced to O (m * 
> logn).
> Another improvement is we do not need to scan every server for each rsgroup. 
> If the processed server could be recorded,  we could skip those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-15 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400397#comment-16400397
 ] 

Hudson commented on HBASE-20186:


Results for branch master
[build #262 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/262/]: (x) 
*{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/262//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/262//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/262//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when 
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
>  Issue Type: Improvement
>  Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-20186.master.000.patch, 
> HBASE-20186.master.001.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map 
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions 
> which have been already placed according to the rsgroup assignment, and it is 
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite 
> efficient, as it does not make good use of the fact that servers in 
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
>   for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
>   groupClusterState.put(curr, correctedState.get(curr));
> }
>   }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each 
> rsgroup in average, the code above has time complexity as O(m * n), while 
> using TreeSet's contains(), the time complexity could be reduced to O (m * 
> logn).
> Another improvement is we do not need to scan every server for each rsgroup. 
> If the processed server could be recorded,  we could skip those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-14 Thread Xiang Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399953#comment-16399953
 ] 

Xiang Li commented on HBASE-20186:
--

Thanks Ted for the review!

> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when 
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
>  Issue Type: Improvement
>  Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-20186.master.000.patch, 
> HBASE-20186.master.001.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map 
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions 
> which have been already placed according to the rsgroup assignment, and it is 
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite 
> efficient, as it does not make good use of the fact that servers in 
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
>   for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
>   groupClusterState.put(curr, correctedState.get(curr));
> }
>   }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each 
> rsgroup in average, the code above has time complexity as O(m * n), while 
> using TreeSet's contains(), the time complexity could be reduced to O (m * 
> logn).
> Another improvement is we do not need to scan every server for each rsgroup. 
> If the processed server could be recorded,  we could skip those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-14 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398514#comment-16398514
 ] 

Hadoop QA commented on HBASE-20186:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
 5s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  5m 
 0s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
19m 27s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
8s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
 9s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20186 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12914401/HBASE-20186.master.001.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux ea15ab317203 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 5650853830 |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11949/testReport/ |
| Max. process+thread count | 2707 (vs. ulimit of 1) |
| modules | C: hbase-rsgroup U: hbase-rsgroup |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11949/console |
| 

[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-13 Thread Xiang Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398043#comment-16398043
 ] 

Xiang Li commented on HBASE-20186:
--

Trigger Hadoop QA for patch 001

> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when 
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
>  Issue Type: Improvement
>  Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Attachments: HBASE-20186.master.000.patch, 
> HBASE-20186.master.001.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map 
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions 
> which have been already placed according to the rsgroup assignment, and it is 
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite 
> efficient, as it does not make good use of the fact that servers in 
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
>   for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
>   groupClusterState.put(curr, correctedState.get(curr));
> }
>   }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each 
> rsgroup in average, the code above has time complexity as O(m * n), while 
> using TreeSet's contains(), the time complexity could be reduced to O (m * 
> logn).
> Another improvement is we do not need to scan every server for each rsgroup. 
> If the processed server could be recorded,  we could skip those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-13 Thread Xiang Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398041#comment-16398041
 ] 

Xiang Li commented on HBASE-20186:
--

Thanks [~yuzhih...@gmail.com].
I corrected the format as well as some typos in comments. Please review it at 
your most convenience!

> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when 
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
>  Issue Type: Improvement
>  Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Attachments: HBASE-20186.master.000.patch, 
> HBASE-20186.master.001.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map 
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions 
> which have been already placed according to the rsgroup assignment, and it is 
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite 
> efficient, as it does not make good use of the fact that servers in 
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
>   for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
>   groupClusterState.put(curr, correctedState.get(curr));
> }
>   }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each 
> rsgroup in average, the code above has time complexity as O(m * n), while 
> using TreeSet's contains(), the time complexity could be reduced to O (m * 
> logn).
> Another improvement is we do not need to scan every server for each rsgroup. 
> If the processed server could be recorded,  we could skip those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-13 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397380#comment-16397380
 ] 

Ted Yu commented on HBASE-20186:


Build error was not related:
{code}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-install-plugin:2.5.2:install (default-install) 
on project hbase-thrift: Failed to install metadata 
org.apache.hbase:hbase-thrift:3.0.0-SNAPSHOT/maven-metadata.xml: Could not 
parse metadata 
/home/jenkins/.m2/repository/org/apache/hbase/hbase-thrift/3.0.0-SNAPSHOT/maven-metadata-local.xml:
 in epilog non whitespace content is not allowed but got / (position: END_TAG 
seen ...\n/... @25:2)  -> [Help 1]
{code}
Can you format the code ?
e.g. leave a space between {{if}} and left parenthesis.

Otherwise looks good.

> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when 
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
>  Issue Type: Improvement
>  Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Attachments: HBASE-20186.master.000.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map 
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions 
> which have been already placed according to the rsgroup assignment, and it is 
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite 
> efficient, as it does not make good use of the fact that servers in 
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
>   for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
>   groupClusterState.put(curr, correctedState.get(curr));
> }
>   }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each 
> rsgroup in average, the code above has time complexity as O(m * n), while 
> using TreeSet's contains(), the time complexity could be reduced to O (m * 
> logn).
> Another improvement is we do not need to scan every server for each rsgroup. 
> If the processed server could be recorded,  we could skip those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-13 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397374#comment-16397374
 ] 

Hadoop QA commented on HBASE-20186:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
44s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
46s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red}  7m  
0s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red}  9m  
8s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 
36s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color} 
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
12s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
 7s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m  3s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20186 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12914303/HBASE-20186.master.000.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 37c8b8dceded 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / 72c3d27bf6 |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC3 |
| 

[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-13 Thread Xiang Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397291#comment-16397291
 ] 

Xiang Li commented on HBASE-20186:
--

Upload the very first patch

> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when 
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
>  Issue Type: Improvement
>  Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Attachments: HBASE-20186.master.000.patch
>
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map 
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions 
> which have been already placed according to the rsgroup assignment, and it is 
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite 
> efficient, as it does not make good use of the fact that servers in 
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
>   for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
>   groupClusterState.put(curr, correctedState.get(curr));
> }
>   }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each 
> rsgroup in average, the code above has time complexity as O(m * n), while 
> using TreeSet's contains(), the time complexity could be reduced to O (m * 
> logn).
> Another improvement is we do not need to scan every server for each rsgroup. 
> If the processed server could be recorded,  we could skip those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20186) Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup

2018-03-13 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397127#comment-16397127
 ] 

Ted Yu commented on HBASE-20186:


bq. could be reduced to O (m * logn)

Seems to be meaningful speedup.

Looking forward to the patch.

> Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when 
> calculating cluster state for each rsgroup
> --
>
> Key: HBASE-20186
> URL: https://issues.apache.org/jira/browse/HBASE-20186
> Project: HBase
>  Issue Type: Improvement
>  Components: rsgroup
> Environment: In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map 
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions 
> which have been already placed according to the rsgroup assignment, and it is 
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite 
> efficient, as it does not make good use of the fact that servers in 
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
>   for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
>   groupClusterState.put(curr, correctedState.get(curr));
> }
>   }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each 
> rsgroup in average, the code above has time complexity as O(m * n), while 
> using TreeSet's contains(), the time complexity could be reduced to O (m * 
> logn).
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
>
> In RSGroupBasedLoadBalancer
> {code}
> public List balanceCluster(Map 
> clusterState)
> {code}
> The second half of the function is to calculate region move plan for regions 
> which have been already placed according to the rsgroup assignment, and it is 
> calculated one rsgroup after another.
> The following logic to check if a server belongs to the rsgroup is not quite 
> efficient, as it does not make good use of the fact that servers in 
> RSGroupInfo is a TreeSet.
> {code}
> for (Address sName : info.getServers()) {
>   for(ServerName curr: clusterState.keySet()) {
> if(curr.getAddress().equals(sName)) {
>   groupClusterState.put(curr, correctedState.get(curr));
> }
>   }
> }
> {code}
> Given there are m region servers in the cluster and n region servers for each 
> rsgroup in average, the code above has time complexity as O(m * n), while 
> using TreeSet's contains(), the time complexity could be reduced to O (m * 
> logn).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)