[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13943711#comment-13943711 ] Phabricator commented on HBASE-5959: eclark has abandoned the revision "HBASE-5959 [jira] Add other load balancers". REVISION DETAIL https://reviews.facebook.net/D3189 To: JIRA, eclark Cc: tedyu, lhofhansl > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.95.2 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.95.2 > > Attachments: ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.1.patch, > ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.2.patch, > ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.3.patch, > ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.4.patch, > ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.5.patch, > ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.6.patch, > ASF.LICENSE.NOT.GRANTED--HBASE-5959.D3189.7.patch, HBASE-5959-0.patch, > HBASE-5959-1.patch, HBASE-5959-11.patch, HBASE-5959-12.patch, > HBASE-5959-13.patch, HBASE-5959-14.patch, HBASE-5959-2.patch, > HBASE-5959-3.patch, HBASE-5959-6.patch, HBASE-5959-7.patch, > HBASE-5959-8.patch, HBASE-5959-9.patch > > > Now that balancers are pluggable we should give some options. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286212#comment-13286212 ] Hudson commented on HBASE-5959: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #33 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/33/]) HBASE-5959 Add other load balancers (Revision 1344457) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/pom.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java * /hbase/trunk/pom.xml > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-13.patch, > HBASE-5959-14.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, > HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch, > HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, > HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286083#comment-13286083 ] Hudson commented on HBASE-5959: --- Integrated in HBase-TRUNK #2957 (See [https://builds.apache.org/job/HBase-TRUNK/2957/]) HBASE-5959 Add other load balancers (Revision 1344457) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/pom.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java * /hbase/trunk/pom.xml > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-13.patch, > HBASE-5959-14.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, > HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch, > HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, > HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286066#comment-13286066 ] Hadoop QA commented on HBASE-5959: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530263/HBASE-5959-14.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2062//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2062//console This message is automatically generated. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-13.patch, > HBASE-5959-14.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, > HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch, > HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, > HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286053#comment-13286053 ] Elliott Clark commented on HBASE-5959: -- done > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-13.patch, > HBASE-5959-14.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, > HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch, > HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, > HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286034#comment-13286034 ] stack commented on HBASE-5959: -- @Ted np Sorry about that. I'll go ahead and commit this. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-13.patch, > HBASE-5959-14.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, > HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch, > HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, > HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286023#comment-13286023 ] Zhihong Yu commented on HBASE-5959: --- The test failure was due to HBASE-5936. I updated workspace and it passed. Let's see what Hadoop QA tells us: https://builds.apache.org/job/PreCommit-HBASE-Build/2062/console > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-13.patch, > HBASE-5959-14.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, > HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch, > HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, > HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286016#comment-13286016
]
Elliott Clark commented on HBASE-5959:
--
Just tried it and got a passing test:
{code}
mvn clean compile test -PrunAllTests -DfailIfNoTests=false
-Dtest=TestLogRollingNoCluster
ruby-1.9.3-p194 [33s]
[INFO] Scanning for projects...
[INFO]
[INFO] Reactor Build Order:
[INFO]
[INFO] HBase
[INFO] HBase - Server
[INFO] HBase - Assembly
[INFO] HBase - Site
[INFO]
[INFO]
[INFO] Building HBase 0.95-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hbase ---
[INFO] Deleting /Users/eclark/Code/hbase/target
[INFO]
[INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase ---
[INFO] Setting property: classpath.resource.loader.class =>
'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
[INFO] Setting property: velocimacro.messages.on => 'false'.
[INFO] Setting property: resource.loader => 'classpath'.
[INFO] Setting property: resource.manager.logwhenfound => 'false'.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase ---
[INFO]
[INFO]
[INFO] Building HBase - Server 0.95-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hbase-server ---
[INFO] Deleting /Users/eclark/Code/hbase/hbase-server/target
[INFO]
[INFO] --- avro-maven-plugin:1.5.3:schema (generate-avro-sources) @
hbase-server ---
[INFO]
[INFO] --- avro-maven-plugin:1.5.3:protocol (generate-avro-sources) @
hbase-server ---
[INFO]
[INFO] --- maven-antrun-plugin:1.6:run (generate) @ hbase-server ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/Users/eclark/Code/hbase/hbase-server/target/hbase-webapps
[copy] Copying 4 files to
/Users/eclark/Code/hbase/hbase-server/target/hbase-webapps
12/05/30 13:36:56 INFO mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
[mkdir] Created dir:
/Users/eclark/Code/hbase/hbase-server/target/hbase-webapps/master/WEB-INF
[mkdir] Created dir:
/Users/eclark/Code/hbase/hbase-server/target/hbase-webapps/regionserver/WEB-INF
[INFO] Executed tasks
[INFO]
[INFO] --- build-helper-maven-plugin:1.5:add-source
(jspcSource-packageInfo-Avro-source) @ hbase-server ---
[INFO] Source directory:
/Users/eclark/Code/hbase/hbase-server/target/generated-jamon added.
[INFO] Source directory:
/Users/eclark/Code/hbase/hbase-server/target/generated-sources/java added.
[INFO]
[INFO] --- jamon-maven-plugin:2.3.4:translate (default) @ hbase-server ---
[INFO] Translating 4 templates from
/Users/eclark/Code/hbase/hbase-server/src/main/jamon to
/Users/eclark/Code/hbase/hbase-server/target/generated-jamon
[INFO]
[INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase-server
---
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @
hbase-server ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @ hbase-server
---
[INFO] Compiling 766 source files to
/Users/eclark/Code/hbase/hbase-server/target/classes
[INFO]
[INFO] --- avro-maven-plugin:1.5.3:schema (generate-avro-sources) @
hbase-server ---
[INFO]
[INFO] --- avro-maven-plugin:1.5.3:protocol (generate-avro-sources) @
hbase-server ---
[INFO]
[INFO] --- maven-antrun-plugin:1.6:run (generate) @ hbase-server ---
[INFO] Executing tasks
main:
12/05/30 13:37:19 INFO mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
[INFO] Executed tasks
[INFO]
[INFO] --- build-helper-maven-plugin:1.5:add-source
(jspcSource-packageInfo-Avro-source) @ hbase-server ---
[INFO] Source directory:
/Users/eclark/Code/hbase/hbase-server/target/generated-jamon added.
[INFO] Source directory:
/Users/eclark/Code/hbase/hbase-server/target/generated-sources/java added.
[INFO]
[INFO] --- jamon-maven-plugin:2.3.4:translate (default) @ hbase-server ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.1:process (default) @ hbase-server
---
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @
hbase-server ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to cop
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286015#comment-13286015 ] stack commented on HBASE-5959: -- @Ted If you run w/o this patch, that test passes and then if you run w/ this patch it always fails? That test seems totally unrelated to whats going on in this patch. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-13.patch, > HBASE-5959-14.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, > HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch, > HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, > HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286012#comment-13286012
]
Zhihong Yu commented on HBASE-5959:
---
I got the following when running against patch v14:
{code}
testContendedLogRolling(org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster)
Time elapsed: 0.201 sec <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertFalse(Assert.java:68)
at org.junit.Assert.assertFalse(Assert.java:79)
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster.testContendedLogRolling(TestLogRollingNoCluster.java:78)
{code}
> Add other load balancers
>
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
> Issue Type: New Feature
> Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch,
> HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-13.patch,
> HBASE-5959-14.patch, HBASE-5959-2.patch, HBASE-5959-3.patch,
> HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch,
> HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch,
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch,
> HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch
>
>
> Now that balancers are pluggable we should give some options.b
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285961#comment-13285961
]
Zhihong Yu commented on HBASE-5959:
---
>From https://builds.apache.org/job/PreCommit-HBASE-Build/2060/console :
{code}
/home/jenkins/tools/maven/latest/bin/mvn clean test -DskipTests
-DHBasePatchProcess >
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkJavacWarnings.txt
2>&1
Trunk compilation is broken?
{code}
> Add other load balancers
>
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
> Issue Type: New Feature
> Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch,
> HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-13.patch,
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch,
> HBASE-5959-7.patch, HBASE-5959-8.patch, HBASE-5959-9.patch,
> HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch,
> HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch,
> HBASE-5959.D3189.7.patch
>
>
> Now that balancers are pluggable we should give some options.b
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285941#comment-13285941 ] stack commented on HBASE-5959: -- I'll commit this this afternoon unless objection (old behavior is the default). > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-13.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959-8.patch, HBASE-5959-9.patch, > HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, > HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch, > HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285302#comment-13285302 ] Zhihong Yu commented on HBASE-5959: --- I was trying to see the difference between Diff 10 and Diff 11. Due to rebasing, the diff is not easy to read. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-2.patch, > HBASE-5959-3.patch, HBASE-5959-6.patch, HBASE-5959-7.patch, > HBASE-5959-8.patch, HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, > HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, > HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285219#comment-13285219 ] stack commented on HBASE-5959: -- I'm +1 on this patch. Ted? > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-12.patch, HBASE-5959-2.patch, > HBASE-5959-3.patch, HBASE-5959-6.patch, HBASE-5959-7.patch, > HBASE-5959-8.patch, HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, > HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, > HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285204#comment-13285204 ] Elliott Clark commented on HBASE-5959: -- bq. Shouldn't setMasterServices instead be passed to the constructor? Otherwise, could have an instance of a balancer that was not viable, not until after setMasterServices was called (ditto w/ setClusterStatus)? It would be nice but using the hadoop ReflectionUtils a conf is the only thing that can be passed in. The balancer should work without either of those, though they are always set on finish master initialization. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, > HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch, > HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, > HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285193#comment-13285193 ] Hadoop QA commented on HBASE-5959: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530097/HBASE-5959-11.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2035//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2035//console This message is automatically generated. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, > HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch, > HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, > HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285189#comment-13285189 ] Elliott Clark commented on HBASE-5959: -- bq. Why does RegionPlanComparator implemenent Serializable? Findbugs wants it so that any collection using this can also be serializable. Probably not needed but trying to correct findbugs where I can. bq. Maybe doc why you are doing this compare on regionid... are you dependent on it being timestamp? bq. Why not do 'return r.getRegionInfo().getRegionId() - l.getRegionInfo().getRegionId();' rather than check < 0 and return -1, etc? These are both just moved code from the old balancer. My bet would be the comparator was written that way so the return works without a type cast and if the diff is greater than an int can hold. But not sure. bq. Should RegionPlanComparator be public? Could it be package private? It needs to be public so the balancer package has access? Yep it needs to be used by the balancer package. bq. Comment seems wrong '+ // The cache for where load balancers are located.' bq. This is an expensive call? computeHDFSBlocksDistribution? Goes to NN? We do it for every region in cluster every time we balance? We doing to DOS the NN? We do this for every region. However we cache this for a given amount of time. Should we up that beyond 30 mins ? bq. Why ServerAndLoad implement Serializable? You planning on java serializing it? Another findbugs special bq. Does StochasticLoadBalancer have to be a public class? It makes the factory much easier. Not 100% required but makes it use less reflection to create an instance. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, > HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch, > HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, > HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285170#comment-13285170 ] stack commented on HBASE-5959: -- Why does RegionPlanComparator implemenent Serializable? Maybe doc why you are doing this compare on regionid... are you dependent on it being timestamp? Why not do 'return r.getRegionInfo().getRegionId() - l.getRegionInfo().getRegionId();' rather than check < 0 and return -1, etc? Remove a 'the' from "...It provides the the functions used to by..." Should RegionPlanComparator be public? Could it be package private? It needs to be public so the balancer package has access? Shouldn't setMasterServices instead be passed to the constructor? Otherwise, could have an instance of a balancer that was not viable, not until after setMasterServices was called (ditto w/ setClusterStatus)? So the method roundRobinAssignment is in the base load balancer. Its expected by all load balancer implementations? Its behavior will be same for all? Fix 'randmo server.' Does ClusterLoadState need to be public? Does it need audience annotation? Why a RegionInfoComparator class and then also a comparator in RegionInfo class? Is the latter a mistaken inclusion? Comment seems wrong '+ // The cache for where load balancers are located.' This is an expensive call? computeHDFSBlocksDistribution? Goes to NN? We do it for every region in cluster every time we balance? We doing to DOS the NN? The call to getTableDescriptor should be getting a descriptor from cache of table descriptors maintained by the master, right? Why ServerAndLoad implement Serializable? You planning on java serializing it? Does StochasticLoadBalancer have to be a public class? This stuff looks great. Nice tests. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-11.patch, HBASE-5959-2.patch, HBASE-5959-3.patch, > HBASE-5959-6.patch, HBASE-5959-7.patch, HBASE-5959-8.patch, > HBASE-5959-9.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, > HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284348#comment-13284348 ] Elliott Clark commented on HBASE-5959: -- Right now the two most variable(read and write) are not used they have a 0 co-efficient. I'll need to add in a lot more plumbing to get a moving average going and this patch is getting a little too long in the tooth as it is. They are the whole point (write requests were the original impetus for this patch). Every time someone talks about load balancing they have a different need. That need is not encompassed by one metric or one single best fit. If it was then the default loadbalnacer would be enough. Tuning this is a pretty difficult challenge (if the defaults are deviated from), however with the complexity comes a more powerful ability to move read/write/memory load. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959-8.patch, HBASE-5959-9.patch, > HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, > HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch, > HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284022#comment-13284022 ] Zhihong Yu commented on HBASE-5959: --- The latest addition of region load costs is limited by the fact that region load currently doesn't reflect moving average. I feel that the 4 new multipliers (readRequestMultiplier, etc) make tuning load balancing harder because there're so many moving parts. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959-8.patch, HBASE-5959-9.patch, > HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, > HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch, > HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282107#comment-13282107 ] Zhihong Yu commented on HBASE-5959: --- I will go over the latest patch one more time. Please allow one or two days. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959-8.patch, HBASE-5959-9.patch, > HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, > HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch, > HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282002#comment-13282002 ] Hadoop QA commented on HBASE-5959: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528790/HBASE-5959-9.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 35 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.replication.TestMultiSlaveReplication org.apache.hadoop.hbase.replication.TestMasterReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1971//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1971//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1971//console This message is automatically generated. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959-8.patch, HBASE-5959-9.patch, > HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, > HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch, > HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281918#comment-13281918 ] Hadoop QA commented on HBASE-5959: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528782/HBASE-5959-8.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1968//console This message is automatically generated. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959-8.patch, HBASE-5959.D3189.1.patch, > HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, > HBASE-5959.D3189.5.patch, HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278329#comment-13278329 ] Hadoop QA commented on HBASE-5959: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527938/HBASE-5959.D3189.7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1920//console This message is automatically generated. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch, > HBASE-5959.D3189.6.patch, HBASE-5959.D3189.7.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277217#comment-13277217 ] Hadoop QA commented on HBASE-5959: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527726/HBASE-5959.D3189.5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1901//console This message is automatically generated. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277013#comment-13277013 ] Elliott Clark commented on HBASE-5959: -- Yeah; Though I've been testing this with per table turned off as that's where this loadbalancer does it's best, when it has the whole knowledge of everything. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276955#comment-13276955 ] Zhihong Yu commented on HBASE-5959: --- bq. we saw production clusters that were no balancing because the old balancer did not want to move regions around to balance the number of regions per table I think the above was due to lack of per table load balancing in 0.92 > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276942#comment-13276942 ] Phabricator commented on HBASE-5959: eclark has commented on the revision "HBASE-5959 [jira] Add other load balancers". I'll get the config storing in the next version INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:241 There are already config paramaters with balancer. we should keep the hierarchy. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:256 And I actually think it should be given a lower weight as the table cost is something that represents an on going cost to the cluster and the locality cost is a one time transfer. The impetus for doing this work was that we saw production clusters that were no balancing because the old balancer did not want to move regions around to balance the number of regions per table (Something I've seen on several different production clusters now; a real issue on anything that has big batch jobs). Placing locality's weight above table cost would just mean that the same thing could happen. However I think the middle ground combined with the ability to change per install is enough without testing on production clusters. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:251 Nope I'll add that in my next version. REVISION DETAIL https://reviews.facebook.net/D3189 To: JIRA, eclark Cc: tedyu > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276923#comment-13276923 ] Phabricator commented on HBASE-5959: tedyu has commented on the revision "HBASE-5959 [jira] Add other load balancers". More comments to follow. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:251 Do we need to re-fetch these config parameters in each iteration ? src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:44 There're extraneous empty lines such as this one. Please remove them. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:78 'one cluster' -> 'cluster with one server' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:80 "it's" -> "cluster has" src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:197 'use' -> 'uses' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:202 Please add javadoc src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:204 Typo: 'pickRandmoRegion' -> 'pickRandomRegion' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:144 Remove empty line. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:154 'plan' -> 'plans' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:224 Please finish the sentence. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:226 Specify what is returned. src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:241 'balancer.stochastic' -> 'stochastic.balancer' src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java:256 I think locality cost should be given higher weight. REVISION DETAIL https://reviews.facebook.net/D3189 To: JIRA, eclark Cc: tedyu > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276317#comment-13276317 ] Elliott Clark commented on HBASE-5959: -- We would have to check sys time every loop through and that would require nanotime to have any real accuracy; nannotime is very expensive. There is currently no soft realtime requirement. It doesn't seem like adding that would really be worth the extra complexity until there is a big need or lots of people asking for it. Also Stack didn't want me to change any of the public interfaces in this issue. If it becomes a big thing we can file an issue and add soft realtime. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276298#comment-13276298 ] Zhihong Yu commented on HBASE-5959: --- bq. need that info in that format. Please add such information. Of particular interest is the timing information of the first several balancing rounds when block location cache is cold. bq. we can't really estimate the amount of time an iteration will take. Agreed. bq. Hence we would have to go back to the timer every iteration through. That would be very costly. Since the maximum time allowable in one iteration is passed to balancer, why would using local timer to gauge elapsed time be expensive ? > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276291#comment-13276291 ] Elliott Clark commented on HBASE-5959: -- The new balancer doesn't have that logging. So there's no timing in the logs. I can add that if there are people that need that info in that format. No since we iterate through all servers there is no given amount of time. It could be that an iteration could: * hit a cold hfile location cache * choose not to move regions. * or do a move. All of those things will take a different amount of time; so we can't really estimate the amount of time an iteration will take. Hence we would have to go back to the timer every iteration through. That would be very costly. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276280#comment-13276280
]
Zhihong Yu commented on HBASE-5959:
---
I was looking for output in master log similar to the following:
{code}
LOG.info("Done. Calculated a load balance in " + (endTime-startTime) + "ms.
" +
"Moving " + totalNumMoved + " regions off of " +
serversOverloaded + " overloaded servers onto " +
serversUnderloaded + " less loaded servers. regions to return:" +
regionsToReturn.size());
{code}
w.r.t. maxSteps, is it possible to deduce the maximum time allowable in one
round of stochastic balancing ?
Currently we have (in HMaster.balance()):
{code}
int maximumBalanceTime = getBalancerCutoffTime();
long cutoffTime = System.currentTimeMillis() + maximumBalanceTime;
{code}
I am thinking of the following:
{code}
int maxPlanGenerationTime = maximumBalanceTime *
// the maximum percentage for planning region rebalance
getConfiguration().getInt("hbase.balancer.max.planning.percent", 30) / 100;
...
List partialPlans =
this.balancer.balanceCluster(assignments,
this.assignmentManager.getTableInitRegions(),
maxPlanGenerationTime);
{code}
> Add other load balancers
>
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
> Issue Type: New Feature
> Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch,
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch,
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch,
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276271#comment-13276271 ] Elliott Clark commented on HBASE-5959: -- hbase(main):004:0> status 13 servers, 0 dead, 3.6154 average load hbase(main):005:0> balancer true 0 row(s) in 3.3460 seconds hbase(main):006:0> balancer true 0 row(s) in 3.3530 seconds hbase(main):007:0> balancer true 0 row(s) in 3.3160 seconds hbase(main):008:0> balancer true 0 row(s) in 3.3850 seconds hbase(main):009:0> balancer true 0 row(s) in 3.3580 seconds hbase(main):010:0> balancer true 0 row(s) in 3.3190 seconds The maxSteps config is nice as it places an upper bounds on the time a large cluster will run the rebalance function. It also allows people to slowly balance their cluster meaning that fewer regions will be moved at a time as the cluster approaches the final solution. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276257#comment-13276257 ] Zhihong Yu commented on HBASE-5959: --- Allow me some time to comment on phabricator. Meanwhile, do you have performance numbers for stochastic balancer on a real cluster ? e.g. how long it takes to balance N regions, etc. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276254#comment-13276254 ] Elliott Clark commented on HBASE-5959: -- And I didn't move the getTopBlockLocations into a helper as this is already a big enough patch and I wanted to keep the impact to a minimum. If this gets accepted i'll be filing tickets to add region load cost function (region load seems broken right now i'm looking into it), and to refactor getTopBlockLocations. Assuming the highest cost or the lowest cost for things has no functional difference. However when/if region load cost functions are added assuming the worst would not work, so I think it's better to start now with always assuming the best cost unless it can be proven otherwise. Other than that yes I believe all of the issues were addressed by what is up on phabricator. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276215#comment-13276215 ] Zhihong Yu commented on HBASE-5959: --- @Elliot: Were all review comments addressed ? Feel free to enumerate the ones you have different opinion with. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276191#comment-13276191 ] Hadoop QA commented on HBASE-5959: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527497/HBASE-5959-7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 33 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1876//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1876//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1876//console This message is automatically generated. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, > HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276093#comment-13276093 ] Hadoop QA commented on HBASE-5959: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527489/HBASE-5959.D3189.4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1874//console This message is automatically generated. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, > HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276092#comment-13276092 ] Hadoop QA commented on HBASE-5959: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527489/HBASE-5959.D3189.4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1873//console This message is automatically generated. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, > HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, > HBASE-5959.D3189.4.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274442#comment-13274442 ] Zhihong Yu commented on HBASE-5959: --- I didn't take into account the scale() call at the end of computeMoveCost(). I would think data locality cost should be given higher weight. Can computeSkewLoadCost() and computeTableSkewLoadCost() be combined ? In default load balancer, regions are balanced per table. The way computeTableSkewLoadCost() currently is implemented, one region from a table on region server can satisfy its goal. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274435#comment-13274435 ] Elliott Clark commented on HBASE-5959: -- Move cost is just there to stop thrashing. We want all the other to be way above what move cost is. If moving were at the same magnitude that regionSkewCost was then the balancer would not work at all. It would only keep the random mutations. Since everything is normalized to between 0 and 1 any multiplier(2 or above) could work, but 100 just makes things a little bit easier to reason about. Basically we'll keep a move if it makes the region skew cost better. And then the balancer will search for how to make that move have the best table skew or the best data locality . > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274432#comment-13274432
]
Zhihong Yu commented on HBASE-5959:
---
{code}
+double moveCost = computeMoveCost(initialRegionMapping, clusterState);
+double regionCountSkewCost = 100 * computeSkewLoadCost(clusterState);
+double tableSkewCost = 50 * computeTableSkewLoadCost(clusterState);
+double localityCost = 40 * computeDataLocalityCost(clusterState);
{code}
Since the value of move cost varies greatly, the coefficients should be
designed such that the three latter costs are brought to the same magnitude of
move cost. Meaning, constant coefficients may not be the best choice.
> Add other load balancers
>
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
> Issue Type: New Feature
> Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch,
> HBASE-5959-2.patch, HBASE-5959-3.patch
>
>
> Now that balancers are pluggable we should give some options.b
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274057#comment-13274057 ] Elliott Clark commented on HBASE-5959: -- I'll put the next version up on a review board. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274041#comment-13274041
]
Zhihong Yu commented on HBASE-5959:
---
For computeTableSkewLoadCost(), normally we use region count (per table) for
region servers to judge skew.
I feel the skew in table count is not a good indicator.
{code}
+ * @return Cost of imbalance in table.
{code}
'table' -> 'tables'
{code}
+for (Entry> entry : clusterState.entrySet())
{
+ Set tablesOnRegionServer = new HashSet();
{code}
We can clear the HashSet after the following line in the loop (instead of
creating HashSet every iteration):
{code}
+ tablesPerServer.add(new Double(tablesOnRegionServer.size()));
{code}
Math.abs() is not needed in the following:
{code}
+ skewCost += Math.abs(numTables - count.doubleValue());
{code}
For computeDataLocalityCost():
{code}
+// If we can't find where the data is getTopBlock returns null.
+// so count that as being the best possible.
+if (dataOnServers == null) {
{code}
I think worst should be assumed if block locations cannot be retrieved.
I asked the coefficients of various costs to be configurable. This is because
users tend to give high priority to locality cost component.
BalancerTestBase.java is missing license.
{code}
+ * Class used to be the base of unit tests on load balancers. It gives helper
{code}
'Class used to be the base' -> 'Base class'.
assertClusterAsBalanced() should be package-private.
For reconcile(),
{code}
+ * This assumes the RegionPlan HSI instances are the same ones in the map, so
+ * actually no need to even pass in the map, but I think it's clearer.
{code}
Please expand HSI to full class name. The two parameters are both Lists. Where
is the map ? I think it is Map map.
Please describe what this method does.
> Add other load balancers
>
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
> Issue Type: New Feature
> Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch,
> HBASE-5959-2.patch, HBASE-5959-3.patch
>
>
> Now that balancers are pluggable we should give some options.b
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273912#comment-13273912
]
Zhihong Yu commented on HBASE-5959:
---
For a patch of this size, review board should be used.
Below is part 1 of my review.
It is necessary to see the benefit of stochastic balancer in a real cluster
compared with existing balancer.
TestRegionRebalancing is an important test which currently exercises default
balancer. It would be nice to apply new balancer there - or create a similar
test class.
{code}
+ * {@link AssignmentManager} to assign regions in the edge cases. It doesn't
{code}
Are edge cases the only ones covered by BaseLoadBalancer ?
{code}
+public class ClusterLoadState {
{code}
Please add javadoc for the above class.
{code}
+ private final NavigableMap> serversByLoad;
{code}
The above map doesn't contain region load information. Can region load be added
?
{code}
+ protected List getInternalTopBlockLocations( HRegionInfo region)
{
{code}
A better name for the above method would be internalGetTopBlockLocations().
Remove the space after '('.
{code}
+ * @param fs
+ * the filesystem
{code}
There is no need to put parameter explanation on separate line.
There're two almost identical getTableDescriptor() methods in master package:
{code}
private HTableDescriptor getTableDescriptor(byte[] tableName)
src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
private HTableDescriptor getTableDescriptor(byte[] tableName)
src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
{code}
It would be nice to put it in a utility class.
For StochasticLoadBalancer:
{code}
+ * randomly try and mutate up to the cluster to Cprime. If F(Cprime) < F(C)
then
{code}
Remove 'up to' above.
{code}
+ public void setMasterServices(MasterServices masterServices) {
+this.services = masterServices;
+ }
{code}
RegionLocationFinder has MasterServices member which should be set in the above
method.
{code}
+ * should always approach
{code}
Please finish the above sentence for balanceCluster().
{code}
+// No need to balance one cluster.
+if (clusterState.size() <= 1) {
{code}
The above check should be placed at the beginning of the method. 'one cluster'
-> 'cluster with one server'
{code}
+// Perform a stocastic walk to see if we can get a good fit.
{code}
'stocastic' -> 'stochastic'
{code}
+HRegionInfo lRegion = pickRandmoRegion(leftRegionList, 0);
+HRegionInfo rRegion = pickRandmoRegion(rightRegionList, 0.5);
{code}
Please explain why selection of move is not symmetrical.
{code}
+boolean keep = RANDOM.nextFloat() < (0.01 * (1 - scale(0, maxSteps,
stepNum)));
+return keep;
{code}
A single line should be enough - return directly.
{code}
+ * @return
+ */
+ private Map createRegionMapping(
{code}
Please add javadoc for return value.
{code}
+double regionCountSkewCost = 100 * computeSkewLoadCost(clusterState);
+double tableSkewCost = 50 * computeTableSkewLoadCost(clusterState);
+double localityCost = 40 * computeDataLocalityCost(clusterState);
{code}
May I ask how the above coefficients were determined ? Should user be able to
change them through config parameters ?
For computeSkewLoadCost():
{code}
+double max = (mean * (clusterState.size() - 1)) + (Math.abs(mean -
numRegions));
{code}
numRegions is total number of regions, meaning it is greater than mean. The
Math.abs() isn't necessary above.
May I ask how the above max is derived ?
> Add other load balancers
>
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
> Issue Type: New Feature
> Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch,
> HBASE-5959-2.patch, HBASE-5959-3.patch
>
>
> Now that balancers are pluggable we should give some options.b
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273765#comment-13273765 ] Hadoop QA commented on HBASE-5959: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526592/HBASE-5959-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 13 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 27 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestDrainingServer Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1855//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1855//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1855//console This message is automatically generated. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch, HBASE-5959-3.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273744#comment-13273744
]
stack commented on HBASE-5959:
--
The move of LoadBalancer down into a package is a no-no given its a Public
Interface (see the annotation just above class declaration). Would need to go
through a deprecation cycle. Factory can go down into the new balancer
package. Thats fine.
You don't need these lines:
{code}
+ * Copyright 2012 The Apache Software Foundation
{code}
Is BalanceInfo generically useful or is it useful only to your new balance algo?
Fix 'It provides the the functions used to by'
You don't want to pass in master services on construction? You don't want to
pass cluster status on invocation of the balance function? Rather than use
setters (when setters, no insurance when they will be called)
Need class comment on new class ClusterLoadState. Does it need to be public?
Do methods in this class need to be public? Can they be package protected?
Did you just move the old classes or do you do some changes to them (I did not
review for changes).
RegionInfoComparator seems like a misleading name given what it does? Indent
is not the two spaces used in rest of code base.
How often will we run this block location calcuation? Its expensive. We need
to be careful.
RegionPlanCompartor should be inner class of RegionPlan?
Will do a deeper review later but so far, looks great (need to look at your new
doohickey balancer more)
> Add other load balancers
>
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
> Issue Type: New Feature
> Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch,
> HBASE-5959-2.patch, HBASE-5959-3.patch
>
>
> Now that balancers are pluggable we should give some options.b
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[ https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273702#comment-13273702 ] Hadoop QA commented on HBASE-5959: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526578/HBASE-5959-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 13 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 28 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.TestDrainingServer org.apache.hadoop.hbase.master.TestAssignmentManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1852//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1852//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1852//console This message is automatically generated. > Add other load balancers > > > Key: HBASE-5959 > URL: https://issues.apache.org/jira/browse/HBASE-5959 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, > HBASE-5959-2.patch > > > Now that balancers are pluggable we should give some options.b -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5959) Add other load balancers
[
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271825#comment-13271825
]
Zhihong Yu commented on HBASE-5959:
---
For this feature, design document showing the origin of Stochastic approach
would help reviewers understand better.
{code}
+class BalanceInfo {
+
+ private final int nextRegionForUnload;
{code}
Please what the integer represents.
{code}
+public abstract class BaseLoadBalancer implements LoadBalancer {
{code}
Please add javadoc for the above class.
{code}
+ protected HTableDescriptor getTableDescriptor(byte[] tableName) throws
IOException {
{code}
Why would the above method reside in BaseLoadBalancer ?
StochasticLoadBalancer, it starts to get interesting.
{code}
+// Perform a stocastic walk to see if we can get a
{code}
Please finish the above comment.
{code}
+if (lRegion != null) {
+ rightRegionList.add(lRegion);
+}
+
+double newCost = computerCost(initialRegionMapping, clusterState);
{code}
computerCost() should be named computeCost(). How would the above call utilize
rightRegionList ?
{code}
+if (newCost < currentCost || RANDOM.nextFloat() < 0.01) {
+ currentCost = newCost;
+} else {
{code}
What if newCost is much worse than currentCost but we choose it due to
RANDOM.nextFloat() < 0.01 ?
{code}
+double skewCost = computeSkewLoadCost(clusterState);
+return moveCost + (100 * skewCost);
{code}
Why would skewCost carry such a big weight ?
Please add test for StochasticLoadBalancer.
> Add other load balancers
>
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
> Issue Type: New Feature
> Components: master
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch
>
>
> Now that balancers are pluggable we should give some options.b
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
