I am interested as to what other folks are seeing when running this test
against their clusters. I think my configuration is sound and the load
is fairly realistic. Please correct me if I'm missing something here.
Here are my results from running the load test on a more recent
cluster. This cluster is 13 x 3.0GHz Quad Core Xeon servers with 16GB
of RAM and two 1.5TB disks. One server is the namenode, one is the
zookeeper (1 of 5), one is the hbase master, and the other ten are data
nodes + region servers. Nodes are connected via IP over Infiniband (10Gbps).
[- Configuration -]
Hadoop Version: 0.20.1, r810220
Hadoop Compiled: Tue Sep 1 20:55:56 UTC 2009 by oom
HBase Version 0.20.1, r822817
HBase Compiled Wed Oct 7 11:55:42 PDT 2009, stack
hadoop-env:
# The maximum amount of heap to use, in MB. Default is 1000.
export HADOOP_HEAPSIZE=8000 # namenode
#export HADOOP_HEAPSIZE=2000 # datanode
# Extra Java runtime options. Empty by default.
# export HADOOP_OPTS=-server
export HADOOP_OPTS="$HADOOP_OPTS -XX:+HeapDumpOnOutOfMemoryError
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
hbase-env:
# The maximum amount of heap to use, in MB. Default is 1000.
export HBASE_HEAPSIZE=6000
export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
[- Test -]
Test was started with a newly formatted namenode. Average load during
the test was 2.66. Highest load seen was 12.95 (62.8%wa) with an average
50MB/s network utilization and 0k swap in use.
After running 2 instances with 30 threads each for just under two hours
both instances aborted with "
org.apache.hadoop.hbase.client.NoServerForRegionException: No server
address listed in .META. for region" errors. At this point two region
servers aborted. I can send you the complete set of logs (74M) if you
are interested.
Error Counts:
1 x ZooKeeper session expired
3 x OutOfMemoryError messages were logged on two different regions
servers.
8 x ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Failed
openScanner
554 x ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration
[- Preliminary Status -]
Attribute Name Value Description
HBase Version 0.20.1, r822817 HBase version and svn revision
HBase Compiled Wed Oct 7 11:55:42 PDT 2009, stack When HBase
version was compiled and by whom
Hadoop Version Unknown, rUnknown Hadoop version and svn revision
Hadoop Compiled Unknown, Unknown When Hadoop version was compiled
and by whom
HBase Root Directory hdfs://hfs-027010:8020/hbase Location of
HBase home directory
Load average 14.5 Average number of regions per regionserver.
Naive computation.
Regions On FS 163 Number of regions on FileSystem. Rough count.
Zookeeper Quorum
hfs-027013:2181,hfs-027012:2181,hfs-027011:2181,hfs-027015:2181,hfs-027014:2181
Addresses of all registered ZK servers. For more, see zk dump.
Catalog Tables
Table Description
-ROOT- The -ROOT- table holds references to all .META. regions.
.META. The .META. table holds references to all User Table regions
User Tables
1 table(s) in set.
Table Description
fc_test {NAME => 'fc_test', FAMILIES => [{NAME => 'data', VERSIONS
=> '1', COMPRESSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
Region Servers
Address Start Code Load
hfs-027013:60030 1257291638999 requests=22, regions=15,
usedHeap=2660, maxHeap=5991
hfs-027014:60030 1257291639075 requests=25, regions=15,
usedHeap=2282, maxHeap=5991
hfs-027015:60030 1257291639015 requests=20, regions=15,
usedHeap=1552, maxHeap=5991
hfs-027016:60030 1257291638947 requests=16, regions=16,
usedHeap=2786, maxHeap=5991
hfs-027017:60030 1257291639078 requests=14, regions=14,
usedHeap=2775, maxHeap=5991
hfs-027018:60030 1257291639284 requests=19, regions=14,
usedHeap=1070, maxHeap=5991
hfs-027019:60030 1257291639149 requests=22, regions=13,
usedHeap=2095, maxHeap=5991
hfs-027020:60030 1257291639289 requests=37, regions=15,
usedHeap=2352, maxHeap=5991
hfs-027021:60030 1257291639067 requests=26, regions=14,
usedHeap=2087, maxHeap=5991
hfs-027022:60030 1257291638995 requests=19, regions=14,
usedHeap=1570, maxHeap=5991
Total: servers: 10 requests=220, regions=145
1154 files and directories, 4417 blocks = 5571 total. Heap Size is 81.06
MB / 7.8 GB (1%)
Configured Capacity : 13.57 TB
DFS Used : 138.83 GB
Non DFS Used : 10.39 GB
DFS Remaining : 13.43 TB
DFS Used% : 1 %
DFS Remaining% : 98.93 %
Live Nodes : 10
Dead Nodes : 0
[- End Status -]
Region Servers
Address Start Code Load
hfs-027013:60030 1257291638999 requests=0, regions=101,
usedHeap=5428, maxHeap=5991
hfs-027014:60030 1257291639075 requests=0, regions=103,
usedHeap=5373, maxHeap=5991
hfs-027015:60030 1257291639015 requests=0, regions=104,
usedHeap=3973, maxHeap=5991
hfs-027016:60030 1257291638947 requests=0, regions=104,
usedHeap=5051, maxHeap=5991
hfs-027017:60030 1257291639078 requests=0, regions=108,
usedHeap=5829, maxHeap=5991
hfs-027018:60030 1257291639284 requests=0, regions=102,
usedHeap=4481, maxHeap=5991
hfs-027019:60030 1257291639149 requests=0, regions=104,
usedHeap=4875, maxHeap=5991
hfs-027020:60030 1257291639289 requests=0, regions=101,
usedHeap=5625, maxHeap=5991
Total: servers: 8 requests=0, regions=827
[- Test Program -]
package org.apache.hadoop.hbase.util;
import java.io.IOException;
import java.io.FileInputStream;
import java.lang.Integer;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
/* create 'fc_test', {NAME => 'json', VERSIONS => 1}, {NAME => 'data',
VERSIONS =>1} */
public class TableTest
extends Thread
{
protected int datasize = 4000000;
public TableTest(int datasize)
{
this.datasize = datasize;
}
public void run()
{
try {
FileInputStream random = new FileInputStream("/dev/urandom");
byte[] key = new byte[32];
byte[] data = new byte[datasize];
HBaseConfiguration hbaseConfiguration = new
HBaseConfiguration();
HTable table = new HTable(hbaseConfiguration, "fc_test");
random.read(data, 0, datasize);
while (true)
{
random.read(key, 0, 32);
Put update = new Put(Bytes.toBytes(getHexString(key)));
update.add(Bytes.toBytes("data"), Bytes.toBytes(""),
data);
table.put(update);
}
} catch (Exception e) {
System.out.println(e.toString());
}
}
public static String getHexString(byte[] b) throws Exception
{
String result = "";
for (int i=0; i < b.length; i++)
{
result +=
Integer.toString( ( b[i] & 0xff ) + 0x100,
16).substring( 1 );
}
return result;
}
public static void main(String args[]) throws IOException
{
Thread t = null;
int i;
int threads = 30;
int datasize = 400000;
try {
threads = Integer.getInteger(args[0]).intValue();
datasize = Integer.getInteger(args[1]).intValue();
} catch (Exception e) {
}
System.out.println("Starting " + threads + " threads");
for (i = 0; i< threads; i++)
{
t = new TableTest(datasize);
t.start();
}
// wait for last thread
try {
t.join();
} catch (InterruptedException e) {
}
}
}
