saintstack commented on a change in pull request #1194: HBASE-23882:
Experimenting with low config settings.
URL: https://github.com/apache/hbase/pull/1194#discussion_r383676582
##########
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
##########
@@ -85,16 +85,69 @@ public static Configuration
addHbaseResources(Configuration conf) {
return conf;
}
+ public static Configuration create() {
+ return create(true);
+ }
+
/**
* Creates a Configuration with HBase resources
* @return a Configuration with HBase resources
*/
- public static Configuration create() {
+ public static Configuration create(boolean lowLimits) {
Configuration conf = new Configuration();
// In case HBaseConfiguration is loaded from a different classloader than
// Configuration, conf needs to be set with appropriate class loader to
resolve
// HBase resources.
conf.setClassLoader(HBaseConfiguration.class.getClassLoader());
+
+
+ if (lowLimits) {
+
+ // can't set this for every test currently
+ // conf.set("hbase.regionserver.hostname", "127.0.0.1");
+
+ conf.setInt("hbase.hfilearchiver.thread.pool.max", 2);
+ conf.setInt("hbase.loadincremental.threads.max", 3);
+
+
+ conf.setInt("hbase.client.sync.wait.timeout.msec", 30000);
+ conf.setInt("zookeeper.recovery.retry", 5);
+ conf.setInt("hbase.client.retries.number", 10);
+ conf.setInt("hbase.hstore.open.and.close.threads.max", 1);
+ conf.setInt(HConstants.REGION_SERVER_REPLICATION_HANDLER_COUNT, 2);
+
+ conf.setInt(HConstants.HBASE_CLIENT_MAX_TOTAL_TASKS, 3);
+
+ conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 2);
+
+
+ conf.setInt(HConstants.REPLICATION_SOURCE_MAXTHREADS_KEY, 1);
+
+ conf.setInt("hbase.netty.worker.count", 1);
+
+ conf.setInt(HConstants.MASTER_OPEN_REGION_THREADS, 3);
+
+ // conf.setInt(HConstants.MASTER_CLOSE_REGION_THREADS, 3);
+ // conf.setInt(HConstants.MASTER_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_META_SERVER_OPERATIONS_THREADS, 2);
+ // conf.setInt(HConstants.MASTER_LOG_REPLAY_OPS_THREADS, 2);
+ // conf.setInt(HConstants.ZOOKEEPER_MAX_CLIENT_CNXNS, 5);
+
+ conf.setInt("hbase.hconnection.threads.max", 30);
Review comment:
Default is 256. I bet a patch w/ just this in it would make a big
difference, what you think?
One though I had was running the tests with fork count of 1 watching the
system for file count, thread count, i/o, memory.... usage as the test suite
ran. I suppose for it to be apples to apples, would need to reuse jvm (but then
tests probably wouldn't pass). Might be good for figuring the heavyweights. You
probably have something that does this already (smile).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
