elsif, i'll give your program a go in a day or so and report back.
St.Ack
On Tue, Nov 3, 2009 at 6:47 PM, elsif <[email protected]> wrote:
> I am interested as to what other folks are seeing when running this test
> against their clusters. I think my configuration is sound and the load
> is fairly realistic. Please correct me if I'm missing something here.
>
> Here are my results from running the load test on a more recent
> cluster. This cluster is 13 x 3.0GHz Quad Core Xeon servers with 16GB
> of RAM and two 1.5TB disks. One server is the namenode, one is the
> zookeeper (1 of 5), one is the hbase master, and the other ten are data
> nodes + region servers. Nodes are connected via IP over Infiniband
> (10Gbps).
>
> [- Configuration -]
>
> Hadoop Version: 0.20.1, r810220
> Hadoop Compiled: Tue Sep 1 20:55:56 UTC 2009 by oom
>
> HBase Version 0.20.1, r822817
> HBase Compiled Wed Oct 7 11:55:42 PDT 2009, stack
>
> hadoop-env:
> # The maximum amount of heap to use, in MB. Default is 1000.
> export HADOOP_HEAPSIZE=8000 # namenode
> #export HADOOP_HEAPSIZE=2000 # datanode
>
> # Extra Java runtime options. Empty by default.
> # export HADOOP_OPTS=-server
> export HADOOP_OPTS="$HADOOP_OPTS -XX:+HeapDumpOnOutOfMemoryError
> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
>
> hbase-env:
> # The maximum amount of heap to use, in MB. Default is 1000.
> export HBASE_HEAPSIZE=6000
>
> export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError
> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
>
>
> [- Test -]
>
> Test was started with a newly formatted namenode. Average load during
> the test was 2.66. Highest load seen was 12.95 (62.8%wa) with an average
> 50MB/s network utilization and 0k swap in use.
>
> After running 2 instances with 30 threads each for just under two hours
> both instances aborted with "
> org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address listed in .META. for region" errors. At this point two region
> servers aborted. I can send you the complete set of logs (74M) if you
> are interested.
>
> Error Counts:
>
> 1 x ZooKeeper session expired
> 3 x OutOfMemoryError messages were logged on two different regions
> servers.
> 8 x ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Failed
> openScanner
> 554 x ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration
>
>
> [- Preliminary Status -]
>
> Attribute Name Value Description
> HBase Version 0.20.1, r822817 HBase version and svn revision
> HBase Compiled Wed Oct 7 11:55:42 PDT 2009, stack When HBase
> version was compiled and by whom
> Hadoop Version Unknown, rUnknown Hadoop version and svn revision
> Hadoop Compiled Unknown, Unknown When Hadoop version was compiled
> and by whom
> HBase Root Directory hdfs://hfs-027010:8020/hbase Location of
> HBase home directory
> Load average 14.5 Average number of regions per regionserver.
> Naive computation.
> Regions On FS 163 Number of regions on FileSystem. Rough count.
> Zookeeper Quorum
>
> hfs-027013:2181,hfs-027012:2181,hfs-027011:2181,hfs-027015:2181,hfs-027014:2181
> Addresses of all registered ZK servers. For more, see zk dump.
> Catalog Tables
> Table Description
> -ROOT- The -ROOT- table holds references to all .META. regions.
> .META. The .META. table holds references to all User Table regions
> User Tables
>
> 1 table(s) in set.
> Table Description
> fc_test {NAME => 'fc_test', FAMILIES => [{NAME => 'data', VERSIONS
> => '1', COMPRESSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536',
> IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
> Region Servers
> Address Start Code Load
> hfs-027013:60030 1257291638999 requests=22, regions=15,
> usedHeap=2660, maxHeap=5991
> hfs-027014:60030 1257291639075 requests=25, regions=15,
> usedHeap=2282, maxHeap=5991
> hfs-027015:60030 1257291639015 requests=20, regions=15,
> usedHeap=1552, maxHeap=5991
> hfs-027016:60030 1257291638947 requests=16, regions=16,
> usedHeap=2786, maxHeap=5991
> hfs-027017:60030 1257291639078 requests=14, regions=14,
> usedHeap=2775, maxHeap=5991
> hfs-027018:60030 1257291639284 requests=19, regions=14,
> usedHeap=1070, maxHeap=5991
> hfs-027019:60030 1257291639149 requests=22, regions=13,
> usedHeap=2095, maxHeap=5991
> hfs-027020:60030 1257291639289 requests=37, regions=15,
> usedHeap=2352, maxHeap=5991
> hfs-027021:60030 1257291639067 requests=26, regions=14,
> usedHeap=2087, maxHeap=5991
> hfs-027022:60030 1257291638995 requests=19, regions=14,
> usedHeap=1570, maxHeap=5991
> Total: servers: 10 requests=220, regions=145
>
>
> 1154 files and directories, 4417 blocks = 5571 total. Heap Size is 81.06
> MB / 7.8 GB (1%)
> Configured Capacity : 13.57 TB
> DFS Used : 138.83 GB
> Non DFS Used : 10.39 GB
> DFS Remaining : 13.43 TB
> DFS Used% : 1 %
> DFS Remaining% : 98.93 %
> Live Nodes : 10
> Dead Nodes : 0
>
> [- End Status -]
> Region Servers
> Address Start Code Load
> hfs-027013:60030 1257291638999 requests=0, regions=101,
> usedHeap=5428, maxHeap=5991
> hfs-027014:60030 1257291639075 requests=0, regions=103,
> usedHeap=5373, maxHeap=5991
> hfs-027015:60030 1257291639015 requests=0, regions=104,
> usedHeap=3973, maxHeap=5991
> hfs-027016:60030 1257291638947 requests=0, regions=104,
> usedHeap=5051, maxHeap=5991
> hfs-027017:60030 1257291639078 requests=0, regions=108,
> usedHeap=5829, maxHeap=5991
> hfs-027018:60030 1257291639284 requests=0, regions=102,
> usedHeap=4481, maxHeap=5991
> hfs-027019:60030 1257291639149 requests=0, regions=104,
> usedHeap=4875, maxHeap=5991
> hfs-027020:60030 1257291639289 requests=0, regions=101,
> usedHeap=5625, maxHeap=5991
> Total: servers: 8 requests=0, regions=827
>
>
> [- Test Program -]
>
> package org.apache.hadoop.hbase.util;
>
> import java.io.IOException;
> import java.io.FileInputStream;
> import java.lang.Integer;
>
> import org.apache.hadoop.hbase.HBaseConfiguration;
> import org.apache.hadoop.hbase.KeyValue;
> import org.apache.hadoop.hbase.client.HTable;
> import org.apache.hadoop.hbase.client.Get;
> import org.apache.hadoop.hbase.client.Put;
> import org.apache.hadoop.hbase.client.Result;
> import org.apache.hadoop.hbase.util.Bytes;
>
> /* create 'fc_test', {NAME => 'json', VERSIONS => 1}, {NAME => 'data',
> VERSIONS =>1} */
>
> public class TableTest
> extends Thread
> {
> protected int datasize = 4000000;
>
> public TableTest(int datasize)
> {
> this.datasize = datasize;
> }
>
> public void run()
> {
> try {
> FileInputStream random = new FileInputStream("/dev/urandom");
> byte[] key = new byte[32];
> byte[] data = new byte[datasize];
>
> HBaseConfiguration hbaseConfiguration = new
> HBaseConfiguration();
> HTable table = new HTable(hbaseConfiguration, "fc_test");
> random.read(data, 0, datasize);
>
> while (true)
> {
> random.read(key, 0, 32);
> Put update = new Put(Bytes.toBytes(getHexString(key)));
> update.add(Bytes.toBytes("data"), Bytes.toBytes(""),
> data);
> table.put(update);
> }
> } catch (Exception e) {
> System.out.println(e.toString());
> }
>
> }
>
> public static String getHexString(byte[] b) throws Exception
> {
> String result = "";
> for (int i=0; i < b.length; i++)
> {
> result +=
> Integer.toString( ( b[i] & 0xff ) + 0x100,
> 16).substring( 1 );
> }
> return result;
> }
>
> public static void main(String args[]) throws IOException
> {
> Thread t = null;
> int i;
>
> int threads = 30;
> int datasize = 400000;
>
> try {
> threads = Integer.getInteger(args[0]).intValue();
> datasize = Integer.getInteger(args[1]).intValue();
> } catch (Exception e) {
> }
>
> System.out.println("Starting " + threads + " threads");
>
> for (i = 0; i< threads; i++)
> {
> t = new TableTest(datasize);
> t.start();
> }
> // wait for last thread
> try {
> t.join();
> } catch (InterruptedException e) {
> }
> }
> }
>
>
