Done (I added you as a contributor in JIRA). Thanks, St.Ack On Tue, Aug 18, 2009 at 12:34 PM, Schubert Zhang <[email protected]> wrote:
> stack, please assign > *HBASE-1778*<https://issues.apache.org/jira/browse/HBASE-1778> to > me. > > On Wed, Aug 19, 2009 at 3:16 AM, Schubert Zhang <[email protected]> wrote: > > > ok, stack. I will done it as soon as possiable. > > > > > > On Wed, Aug 19, 2009 at 2:47 AM, stack <[email protected]> wrote: > > > >> Can you make an issue and a patch please Schubert? > >> St.Ack > >> > >> On Tue, Aug 18, 2009 at 10:52 AM, Schubert Zhang <[email protected]> > >> wrote: > >> > >> > We found that there are two issues about the PerformanceEvaluation > >> class. > >> > - Is not match for hadoop-0.20.0. > >> > - The approach to split map is not strict. Need to provide correct > >> > InputSplit and InputFormat classes. > >> > > >> > And we have just modified the > >> org.apache.hadoop.hbase.PerformanceEvaluation > >> > for our evaluations. Please get our code at: > >> > > >> > http://dl.getdropbox.com/u/24074/code/PerformanceEvaluation.java > >> > > >> > Here is our evaluations of 0.20.0 RC1 > >> > > >> > http://docloud.blogspot.com/2009/08/hbase-0200-performance-evaluation.html > >> > > >> > Schubert > >> > > >> > On Tue, Aug 18, 2009 at 8:37 AM, Jeff Hammerbacher < > [email protected] > >> > >wrote: > >> > > >> > > Thanks guys. For the lazy (e.g. me) and future searchers, here are > >> some > >> > > links. The benchmark is meant to simulate the same performance tests > >> > quoted > >> > > in Google's BigTable paper. > >> > > > >> > > * PerformanceEvaluation wiki page: > >> > > http://wiki.apache.org/hadoop/Hbase/PerformanceEvaluation > >> > > * PerformanceEvaluation.java: > >> > > > >> > > > >> > > >> > http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/test/org/apache/hadoop/hbase/PerformanceEvaluation.java?view=co > >> > > > >> > > Thanks, > >> > > Jeff > >> > > > >> > > On Mon, Aug 17, 2009 at 5:09 PM, stack <[email protected]> wrote: > >> > > > >> > > > On Mon, Aug 17, 2009 at 4:54 PM, Jeff Hammerbacher < > >> > [email protected] > >> > > > >wrote: > >> > > > > >> > > > > Hey Stack, > >> > > > > > >> > > > > I notice that the patch for this issue doesn't include any sort > of > >> > > tests > >> > > > > that might have caught this regression. Do you guys have an > >> > HBaseBench, > >> > > > > HBaseMix, or similarly named tool for catching performance > >> > regressions? > >> > > > > > >> > > > > >> > > > Not as part of our build. The way its currently done is that near > >> > > release, > >> > > > we run our little PerformanceEvaluation doohickey. If its way > off, > >> > crack > >> > > > the profiler. > >> > > > > >> > > > We have been trying to get some of the hadoop allotment of EC2 > time > >> so > >> > we > >> > > > could set up a regular run up on AWS but no luck so far. > >> > > > > >> > > > Good on your Jeff, > >> > > > St.Ack > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > > >> > > > > Thanks, > >> > > > > Jeff > >> > > > > > >> > > > > On Mon, Aug 17, 2009 at 4:51 PM, stack <[email protected]> > wrote: > >> > > > > > >> > > > > > Our writes were off by a factor of 7 or 8. Writes should be > >> better > >> > > now > >> > > > > > (HBASE-1771). > >> > > > > > Thanks, > >> > > > > > St.Ack > >> > > > > > > >> > > > > > > >> > > > > > On Thu, Aug 13, 2009 at 4:53 PM, stack <[email protected]> > >> wrote: > >> > > > > > > >> > > > > > > I just tried it. It seems slow to me writing too. Let me > >> take a > >> > > > > > look.... > >> > > > > > > St.Ack > >> > > > > > > > >> > > > > > > > >> > > > > > > On Thu, Aug 13, 2009 at 10:06 AM, llpind < > >> [email protected] > >> > > > >> > > > > wrote: > >> > > > > > > > >> > > > > > >> > >> > > > > > >> Okay I changed replication to 2. and removed > "-XX:NewSize=6m > >> > > > > > >> -XX:MaxNewSize=6m" > >> > > > > > >> > >> > > > > > >> here is results for randomWrite 3 clients: > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> RandomWrite > ================================================= > >> > > > > > >> > >> > > > > > >> hadoop-0.20.0/bin/hadoop jar > >> hbase-0.20.0/hbase-0.20.0-test.jar > >> > > > > > >> --nomapred > >> > > > > > >> randomWrite 3 > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> 09/08/13 09:51:15 INFO hbase.PerformanceEvaluation: > client-0 > >> > Start > >> > > > > > >> randomWrite at offset 0 for 1048576 rows > >> > > > > > >> 09/08/13 09:51:15 INFO hbase.PerformanceEvaluation: > client-1 > >> > Start > >> > > > > > >> randomWrite at offset 1048576 for 1048576 rows > >> > > > > > >> 09/08/13 09:51:15 INFO hbase.PerformanceEvaluation: > client-2 > >> > Start > >> > > > > > >> randomWrite at offset 2097152 for 1048576 rows > >> > > > > > >> 09/08/13 09:51:47 INFO hbase.PerformanceEvaluation: > client-0 > >> > > > > > >> 0/104857/1048576 > >> > > > > > >> 09/08/13 09:51:48 INFO hbase.PerformanceEvaluation: > client-1 > >> > > > > > >> 1048576/1153427/2097152 > >> > > > > > >> 09/08/13 09:51:48 INFO hbase.PerformanceEvaluation: > client-2 > >> > > > > > >> 2097152/2201997/3145728 > >> > > > > > >> 09/08/13 09:52:22 INFO hbase.PerformanceEvaluation: > client-1 > >> > > > > > >> 1048576/1258284/2097152 > >> > > > > > >> 09/08/13 09:52:23 INFO hbase.PerformanceEvaluation: > client-0 > >> > > > > > >> 0/209714/1048576 > >> > > > > > >> 09/08/13 09:52:24 INFO hbase.PerformanceEvaluation: > client-2 > >> > > > > > >> 2097152/2306854/3145728 > >> > > > > > >> 09/08/13 09:52:47 INFO hbase.PerformanceEvaluation: > client-1 > >> > > > > > >> 1048576/1363141/2097152 > >> > > > > > >> 09/08/13 09:52:58 INFO hbase.PerformanceEvaluation: > client-0 > >> > > > > > >> 0/314571/1048576 > >> > > > > > >> 09/08/13 09:52:58 INFO hbase.PerformanceEvaluation: > client-2 > >> > > > > > >> 2097152/2411711/3145728 > >> > > > > > >> 09/08/13 09:53:24 INFO hbase.PerformanceEvaluation: > client-1 > >> > > > > > >> 1048576/1467998/2097152 > >> > > > > > >> 09/08/13 09:53:27 INFO hbase.PerformanceEvaluation: > client-0 > >> > > > > > >> 0/419428/1048576 > >> > > > > > >> 09/08/13 09:53:27 INFO hbase.PerformanceEvaluation: > client-2 > >> > > > > > >> 2097152/2516568/3145728 > >> > > > > > >> 09/08/13 09:53:48 INFO hbase.PerformanceEvaluation: > client-1 > >> > > > > > >> 1048576/1572855/2097152 > >> > > > > > >> 09/08/13 09:54:08 INFO hbase.PerformanceEvaluation: > client-2 > >> > > > > > >> 2097152/2621425/3145728 > >> > > > > > >> 09/08/13 09:54:10 INFO hbase.PerformanceEvaluation: > client-0 > >> > > > > > >> 0/524285/1048576 > >> > > > > > >> 09/08/13 09:54:40 INFO hbase.PerformanceEvaluation: > client-1 > >> > > > > > >> 1048576/1677712/2097152 > >> > > > > > >> 09/08/13 09:54:49 INFO hbase.PerformanceEvaluation: > client-2 > >> > > > > > >> 2097152/2726282/3145728 > >> > > > > > >> 09/08/13 09:54:52 INFO hbase.PerformanceEvaluation: > client-0 > >> > > > > > >> 0/629142/1048576 > >> > > > > > >> 09/08/13 09:55:57 INFO hbase.PerformanceEvaluation: > client-1 > >> > > > > > >> 1048576/1782569/2097152 > >> > > > > > >> 09/08/13 09:56:21 INFO hbase.PerformanceEvaluation: > client-2 > >> > > > > > >> 2097152/2831139/3145728 > >> > > > > > >> 09/08/13 09:56:41 INFO hbase.PerformanceEvaluation: > client-0 > >> > > > > > >> 0/733999/1048576 > >> > > > > > >> 09/08/13 09:57:23 INFO hbase.PerformanceEvaluation: > client-1 > >> > > > > > >> 1048576/1887426/2097152 > >> > > > > > >> 09/08/13 09:58:40 INFO hbase.PerformanceEvaluation: > client-2 > >> > > > > > >> 2097152/2935996/3145728 > >> > > > > > >> 09/08/13 09:58:54 INFO hbase.PerformanceEvaluation: > client-0 > >> > > > > > >> 0/838856/1048576 > >> > > > > > >> 09/08/13 10:00:29 INFO hbase.PerformanceEvaluation: > client-1 > >> > > > > > >> 1048576/1992283/2097152 > >> > > > > > >> 09/08/13 10:01:01 INFO hbase.PerformanceEvaluation: > client-2 > >> > > > > > >> 2097152/3040853/3145728 > >> > > > > > >> 09/08/13 10:01:24 INFO hbase.PerformanceEvaluation: > client-0 > >> > > > > > >> 0/943713/1048576 > >> > > > > > >> 09/08/13 10:02:36 INFO hbase.PerformanceEvaluation: > client-1 > >> > > > > > >> 1048576/2097140/2097152 > >> > > > > > >> 09/08/13 10:02:37 INFO hbase.PerformanceEvaluation: > client-1 > >> > > > Finished > >> > > > > > >> randomWrite in 680674ms at offset 1048576 for 1048576 rows > >> > > > > > >> 09/08/13 10:02:37 INFO hbase.PerformanceEvaluation: > Finished > >> 1 > >> > in > >> > > > > > 680674ms > >> > > > > > >> writing 1048576 rows > >> > > > > > >> 09/08/13 10:03:19 INFO hbase.PerformanceEvaluation: > client-2 > >> > > > > > >> 2097152/3145710/3145728 > >> > > > > > >> 09/08/13 10:03:20 INFO hbase.PerformanceEvaluation: > client-2 > >> > > > Finished > >> > > > > > >> randomWrite in 723771ms at offset 2097152 for 1048576 rows > >> > > > > > >> 09/08/13 10:03:20 INFO hbase.PerformanceEvaluation: > Finished > >> 2 > >> > in > >> > > > > > 723771ms > >> > > > > > >> writing 1048576 rows > >> > > > > > >> 09/08/13 10:03:41 INFO hbase.PerformanceEvaluation: > client-0 > >> > > > > > >> 0/1048570/1048576 > >> > > > > > >> 09/08/13 10:03:42 INFO hbase.PerformanceEvaluation: > client-0 > >> > > > Finished > >> > > > > > >> randomWrite in 746054ms at offset 0 for 1048576 rows > >> > > > > > >> 09/08/13 10:03:42 INFO hbase.PerformanceEvaluation: > Finished > >> 0 > >> > in > >> > > > > > 746054ms > >> > > > > > >> writing 1048576 rows > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > ============================================================ > >> > > > > > >> > >> > > > > > >> Still pretty slow. Any other ideas? I'm running the > client > >> > from > >> > > > the > >> > > > > > >> master > >> > > > > > >> box, but its not running any regionServers or datanodes. > >> > > > > > >> > >> > > > > > >> stack-3 wrote: > >> > > > > > >> > > >> > > > > > >> > Your config. looks fine. > >> > > > > > >> > > >> > > > > > >> > Only think that gives me pause is: > >> > > > > > >> > > >> > > > > > >> > "-XX:NewSize=6m -XX:MaxNewSize=6m" > >> > > > > > >> > > >> > > > > > >> > Any reason for the above? > >> > > > > > >> > > >> > > > > > >> > If you study your GC logs, lots of pauses? > >> > > > > > >> > > >> > > > > > >> > Oh, and this: replication is set to 6. Why 6? Each > write > >> > must > >> > > > > commit > >> > > > > > >> to > >> > > > > > >> > 6 > >> > > > > > >> > datanodes before complete. In the tests posted on wiki, > we > >> > > > > replicate > >> > > > > > to > >> > > > > > >> 3 > >> > > > > > >> > nodes. > >> > > > > > >> > > >> > > > > > >> > In end of this message you say you are doing gets? > Numbers > >> > you > >> > > > > posted > >> > > > > > >> > were > >> > > > > > >> > for writes? > >> > > > > > >> > > >> > > > > > >> > St.Ack > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > On Wed, Aug 12, 2009 at 1:15 PM, llpind < > >> > [email protected] > >> > > > > >> > > > > > wrote: > >> > > > > > >> > > >> > > > > > >> >> > >> > > > > > >> >> Not sure why my performance is so slow. Here is my > >> > > > configuration: > >> > > > > > >> >> > >> > > > > > >> >> box1: > >> > > > > > >> >> 10395 SecondaryNameNode > >> > > > > > >> >> 11628 Jps > >> > > > > > >> >> 10131 NameNode > >> > > > > > >> >> 10638 HQuorumPeer > >> > > > > > >> >> 10705 HMaster > >> > > > > > >> >> > >> > > > > > >> >> box 2-5: > >> > > > > > >> >> 6741 HQuorumPeer > >> > > > > > >> >> 6841 HRegionServer > >> > > > > > >> >> 7881 Jps > >> > > > > > >> >> 6610 DataNode > >> > > > > > >> >> > >> > > > > > >> >> > >> > > > > > >> >> hbase site: ======================= > >> > > > > > >> >> <?xml version="1.0"?> > >> > > > > > >> >> <?xml-stylesheet type="text/xsl" > >> href="configuration.xsl"?> > >> > > > > > >> >> <!-- > >> > > > > > >> >> /** > >> > > > > > >> >> * Copyright 2007 The Apache Software Foundation > >> > > > > > >> >> * > >> > > > > > >> >> * Licensed to the Apache Software Foundation (ASF) > under > >> one > >> > > > > > >> >> * or more contributor license agreements. See the > NOTICE > >> > file > >> > > > > > >> >> * distributed with this work for additional information > >> > > > > > >> >> * regarding copyright ownership. The ASF licenses this > >> file > >> > > > > > >> >> * to you under the Apache License, Version 2.0 (the > >> > > > > > >> >> * "License"); you may not use this file except in > >> compliance > >> > > > > > >> >> * with the License. You may obtain a copy of the > License > >> at > >> > > > > > >> >> * > >> > > > > > >> >> * http://www.apache.org/licenses/LICENSE-2.0 > >> > > > > > >> >> * > >> > > > > > >> >> * Unless required by applicable law or agreed to in > >> writing, > >> > > > > > software > >> > > > > > >> >> * distributed under the License is distributed on an > "AS > >> IS" > >> > > > > BASIS, > >> > > > > > >> >> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either > >> > express > >> > > > or > >> > > > > > >> >> implied. > >> > > > > > >> >> * See the License for the specific language governing > >> > > > permissions > >> > > > > > and > >> > > > > > >> >> * limitations under the License. > >> > > > > > >> >> */ > >> > > > > > >> >> --> > >> > > > > > >> >> <configuration> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>hbase.rootdir</name> > >> > > > > > >> >> <value>hdfs://box1:9000/hbase</value> > >> > > > > > >> >> <description>The directory shared by region servers. > >> > > > > > >> >> </description> > >> > > > > > >> >> </property> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>hbase.master.port</name> > >> > > > > > >> >> <value>60000</value> > >> > > > > > >> >> <description>The port that the HBase master runs at. > >> > > > > > >> >> </description> > >> > > > > > >> >> </property> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>hbase.cluster.distributed</name> > >> > > > > > >> >> <value>true</value> > >> > > > > > >> >> <description>The mode the cluster will be in. > Possible > >> > > values > >> > > > > are > >> > > > > > >> >> false: standalone and pseudo-distributed setups > with > >> > > managed > >> > > > > > >> >> Zookeeper > >> > > > > > >> >> true: fully-distributed with unmanaged Zookeeper > >> Quorum > >> > > (see > >> > > > > > >> >> hbase-env.sh) > >> > > > > > >> >> </description> > >> > > > > > >> >> </property> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>hbase.regionserver.lease.period</name> > >> > > > > > >> >> <value>120000</value> > >> > > > > > >> >> <description>HRegion server lease period in > >> milliseconds. > >> > > > > Default > >> > > > > > is > >> > > > > > >> >> 60 seconds. Clients must report in within this period > >> else > >> > > > they > >> > > > > > are > >> > > > > > >> >> considered dead.</description> > >> > > > > > >> >> </property> > >> > > > > > >> >> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>hbase.zookeeper.property.clientPort</name> > >> > > > > > >> >> <value>2222</value> > >> > > > > > >> >> <description>Property from ZooKeeper's config > >> zoo.cfg. > >> > > > > > >> >> The port at which the clients will connect. > >> > > > > > >> >> </description> > >> > > > > > >> >> </property> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>hbase.zookeeper.property.dataDir</name> > >> > > > > > >> >> <value>/home/hadoop/zookeeper</value> > >> > > > > > >> >> </property> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>hbase.zookeeper.property.syncLimit</name> > >> > > > > > >> >> <value>5</value> > >> > > > > > >> >> </property> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>hbase.zookeeper.property.tickTime</name> > >> > > > > > >> >> <value>2000</value> > >> > > > > > >> >> </property> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>hbase.zookeeper.property.initLimit</name> > >> > > > > > >> >> <value>10</value> > >> > > > > > >> >> </property> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>hbase.zookeeper.quorum</name> > >> > > > > > >> >> <value>box1,box2,box3,box4</value> > >> > > > > > >> >> <description>Comma separated list of servers in the > >> > > > ZooKeeper > >> > > > > > >> >> Quorum. > >> > > > > > >> >> For example, > >> > > > > > >> >> "host1.mydomain.com,host2.mydomain.com, > host3.mydomain.com > >> ". > >> > > > > > >> >> By default this is set to localhost for local and > >> > > > > > >> pseudo-distributed > >> > > > > > >> >> modes > >> > > > > > >> >> of operation. For a fully-distributed setup, this > >> should > >> > > be > >> > > > > set > >> > > > > > to > >> > > > > > >> a > >> > > > > > >> >> full > >> > > > > > >> >> list of ZooKeeper quorum servers. If > HBASE_MANAGES_ZK > >> is > >> > > set > >> > > > > in > >> > > > > > >> >> hbase-env.sh > >> > > > > > >> >> this is the list of servers which we will > start/stop > >> > > > ZooKeeper > >> > > > > > on. > >> > > > > > >> >> </description> > >> > > > > > >> >> </property> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>hfile.block.cache.size</name> > >> > > > > > >> >> <value>.5</value> > >> > > > > > >> >> <description>text</description> > >> > > > > > >> >> </property> > >> > > > > > >> >> > >> > > > > > >> >> </configuration> > >> > > > > > >> >> > >> > > > > > >> >> > >> > > > > > >> >> hbase > >> > env:==================================================== > >> > > > > > >> >> > >> > > > > > >> >> export HBASE_CLASSPATH=${HADOOP_CONF_DIR} > >> > > > > > >> >> > >> > > > > > >> >> export HBASE_HEAPSIZE=3000 > >> > > > > > >> >> > >> > > > > > >> >> export HBASE_OPTS="-XX:NewSize=6m -XX:MaxNewSize=6m > >> > > > > > >> >> -XX:+UseConcMarkSweepGC > >> > > > > > >> >> -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > >> > > > > > >> >> -XX:+CMSIncrementalMode > >> > > > > > >> >> -Xloggc:/home/hadoop/hbase-0.20.0/logs/gc-hbase.log" > >> > > > > > >> >> > >> > > > > > >> >> export HBASE_MANAGES_ZK=true > >> > > > > > >> >> > >> > > > > > >> >> Hadoop core > >> > > > > > >> >> > >> > site=========================================================== > >> > > > > > >> >> > >> > > > > > >> >> <?xml version="1.0"?> > >> > > > > > >> >> <?xml-stylesheet type="text/xsl" > >> href="configuration.xsl"?> > >> > > > > > >> >> > >> > > > > > >> >> <!-- Put site-specific property overrides in this file. > >> --> > >> > > > > > >> >> > >> > > > > > >> >> <configuration> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>fs.default.name</name> > >> > > > > > >> >> <value>hdfs://box1:9000</value> > >> > > > > > >> >> <description>The name of the default file system. A > URI > >> > > whose > >> > > > > > >> >> scheme and authority determine the FileSystem > >> > implementation. > >> > > > > The > >> > > > > > >> >> uri's scheme determines the config property > >> > (fs.SCHEME.impl) > >> > > > > naming > >> > > > > > >> >> the FileSystem implementation class. The uri's > >> authority > >> > is > >> > > > used > >> > > > > > to > >> > > > > > >> >> determine the host, port, etc. for a > >> > > filesystem.</description> > >> > > > > > >> >> </property> > >> > > > > > >> >> <property> > >> > > > > > >> >> <name>hadoop.tmp.dir</name> > >> > > > > > >> >> <value>/data/hadoop-0.20.0-${user.name}</value> > >> > > > > > >> >> <description>A base for other temporary > >> > > > directories.</description> > >> > > > > > >> >> </property> > >> > > > > > >> >> </configuration> > >> > > > > > >> >> > >> > > > > > >> >> ============== > >> > > > > > >> >> > >> > > > > > >> >> replication is set to 6. > >> > > > > > >> >> > >> > > > > > >> >> hadoop env================= > >> > > > > > >> >> > >> > > > > > >> >> export HADOOP_HEAPSIZE=3000 > >> > > > > > >> >> export > >> HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote > >> > > > > > >> >> $HADOOP_NAMENODE_OPTS" > >> > > > > > >> >> export > >> > > > > HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote > >> > > > > > >> >> $HADOOP_SECONDARYNAMENODE_OPTS" > >> > > > > > >> >> export > >> HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote > >> > > > > > >> >> $HADOOP_DATANODE_OPTS" > >> > > > > > >> >> export > >> HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote > >> > > > > > >> >> $HADOOP_BALANCER_OPTS" > >> > > > > > >> >> export > >> HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote > >> > > > > > >> >> $HADOOP_JOBTRACKER_OPTS" > >> > > > > > >> >> ================== > >> > > > > > >> >> > >> > > > > > >> >> > >> > > > > > >> >> Very basic setup. then i start the cluster do simple > >> random > >> > > Get > >> > > > > > >> >> operations > >> > > > > > >> >> on a tall table (~60 M rows): > >> > > > > > >> >> > >> > > > > > >> >> {NAME => 'tallTable', FAMILIES => [{NAME => 'family1', > >> > > > COMPRESSION > >> > > > > => > >> > > > > > >> >> 'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE > => > >> > > > '65536', > >> > > > > > >> >> IN_MEMORY => 'false', BLOCKCACHE => 'true'}]} > >> > > > > > >> >> > >> > > > > > >> >> Is this fairly normal speeds? I'm unsure if this is a > >> result > >> > > of > >> > > > > > having > >> > > > > > >> a > >> > > > > > >> >> small cluster? Please advise... > >> > > > > > >> >> > >> > > > > > >> >> stack-3 wrote: > >> > > > > > >> >> > > >> > > > > > >> >> > Yeah, seems slow. In old hbase, it could do 5-10k > >> writes a > >> > > > > second > >> > > > > > >> >> going > >> > > > > > >> >> > by > >> > > > > > >> >> > performance eval page up on wiki. SequentialWrite was > >> > about > >> > > > same > >> > > > > > as > >> > > > > > >> >> > RandomWrite. Check out the stats on hw up on that > page > >> and > >> > > > > > >> description > >> > > > > > >> >> of > >> > > > > > >> >> > how test was set up. Can you figure where its slow? > >> > > > > > >> >> > > >> > > > > > >> >> > St.Ack > >> > > > > > >> >> > > >> > > > > > >> >> > On Wed, Aug 12, 2009 at 10:10 AM, llpind < > >> > > > [email protected] > >> > > > > > > >> > > > > > >> >> wrote: > >> > > > > > >> >> > > >> > > > > > >> >> >> > >> > > > > > >> >> >> Thanks Stack. > >> > > > > > >> >> >> > >> > > > > > >> >> >> I will try mapred with more clients. I tried it > >> without > >> > > > mapred > >> > > > > > >> using > >> > > > > > >> >> 3 > >> > > > > > >> >> >> clients Random Write operations here was the output: > >> > > > > > >> >> >> > >> > > > > > >> >> >> 09/08/12 09:22:52 INFO hbase.PerformanceEvaluation: > >> > client-0 > >> > > > > Start > >> > > > > > >> >> >> randomWrite at offset 0 for 1048576 rows > >> > > > > > >> >> >> 09/08/12 09:22:52 INFO hbase.PerformanceEvaluation: > >> > client-1 > >> > > > > Start > >> > > > > > >> >> >> randomWrite at offset 1048576 for 1048576 rows > >> > > > > > >> >> >> 09/08/12 09:22:52 INFO hbase.PerformanceEvaluation: > >> > client-2 > >> > > > > Start > >> > > > > > >> >> >> randomWrite at offset 2097152 for 1048576 rows > >> > > > > > >> >> >> 09/08/12 09:24:23 INFO hbase.PerformanceEvaluation: > >> > client-1 > >> > > > > > >> >> >> 1048576/1153427/2097152 > >> > > > > > >> >> >> 09/08/12 09:24:23 INFO hbase.PerformanceEvaluation: > >> > client-2 > >> > > > > > >> >> >> 2097152/2201997/3145728 > >> > > > > > >> >> >> 09/08/12 09:24:25 INFO hbase.PerformanceEvaluation: > >> > client-0 > >> > > > > > >> >> >> 0/104857/1048576 > >> > > > > > >> >> >> 09/08/12 09:27:42 INFO hbase.PerformanceEvaluation: > >> > client-0 > >> > > > > > >> >> >> 0/209714/1048576 > >> > > > > > >> >> >> 09/08/12 09:27:46 INFO hbase.PerformanceEvaluation: > >> > client-1 > >> > > > > > >> >> >> 1048576/1258284/2097152 > >> > > > > > >> >> >> 09/08/12 09:27:46 INFO hbase.PerformanceEvaluation: > >> > client-2 > >> > > > > > >> >> >> 2097152/2306854/3145728 > >> > > > > > >> >> >> 09/08/12 09:32:32 INFO hbase.PerformanceEvaluation: > >> > client-1 > >> > > > > > >> >> >> 1048576/1363141/2097152 > >> > > > > > >> >> >> 09/08/12 09:32:33 INFO hbase.PerformanceEvaluation: > >> > client-0 > >> > > > > > >> >> >> 0/314571/1048576 > >> > > > > > >> >> >> 09/08/12 09:32:41 INFO hbase.PerformanceEvaluation: > >> > client-2 > >> > > > > > >> >> >> 2097152/2411711/3145728 > >> > > > > > >> >> >> 09/08/12 09:35:31 INFO hbase.PerformanceEvaluation: > >> > client-0 > >> > > > > > >> >> >> 0/419428/1048576 > >> > > > > > >> >> >> 09/08/12 09:35:34 INFO hbase.PerformanceEvaluation: > >> > client-1 > >> > > > > > >> >> >> 1048576/1467998/2097152 > >> > > > > > >> >> >> 09/08/12 09:35:53 INFO hbase.PerformanceEvaluation: > >> > client-2 > >> > > > > > >> >> >> 2097152/2516568/3145728 > >> > > > > > >> >> >> 09/08/12 09:39:02 INFO hbase.PerformanceEvaluation: > >> > client-0 > >> > > > > > >> >> >> 0/524285/1048576 > >> > > > > > >> >> >> 09/08/12 09:39:03 INFO hbase.PerformanceEvaluation: > >> > client-2 > >> > > > > > >> >> >> 2097152/2621425/3145728 > >> > > > > > >> >> >> 09/08/12 09:40:07 INFO hbase.PerformanceEvaluation: > >> > client-1 > >> > > > > > >> >> >> 1048576/1572855/2097152 > >> > > > > > >> >> >> 09/08/12 09:42:53 INFO hbase.PerformanceEvaluation: > >> > client-0 > >> > > > > > >> >> >> 0/629142/1048576 > >> > > > > > >> >> >> 09/08/12 09:44:25 INFO hbase.PerformanceEvaluation: > >> > client-2 > >> > > > > > >> >> >> 2097152/2726282/3145728 > >> > > > > > >> >> >> 09/08/12 09:44:44 INFO hbase.PerformanceEvaluation: > >> > client-1 > >> > > > > > >> >> >> 1048576/1677712/2097152 > >> > > > > > >> >> >> 09/08/12 09:46:43 INFO hbase.PerformanceEvaluation: > >> > client-0 > >> > > > > > >> >> >> 0/733999/1048576 > >> > > > > > >> >> >> 09/08/12 09:48:11 INFO hbase.PerformanceEvaluation: > >> > client-2 > >> > > > > > >> >> >> 2097152/2831139/3145728 > >> > > > > > >> >> >> 09/08/12 09:48:29 INFO hbase.PerformanceEvaluation: > >> > client-1 > >> > > > > > >> >> >> 1048576/1782569/2097152 > >> > > > > > >> >> >> 09/08/12 09:50:12 INFO hbase.PerformanceEvaluation: > >> > client-0 > >> > > > > > >> >> >> 0/838856/1048576 > >> > > > > > >> >> >> 09/08/12 09:52:47 INFO hbase.PerformanceEvaluation: > >> > client-2 > >> > > > > > >> >> >> 2097152/2935996/3145728 > >> > > > > > >> >> >> 09/08/12 09:53:51 INFO hbase.PerformanceEvaluation: > >> > client-1 > >> > > > > > >> >> >> 1048576/1887426/2097152 > >> > > > > > >> >> >> 09/08/12 09:56:32 INFO hbase.PerformanceEvaluation: > >> > client-0 > >> > > > > > >> >> >> 0/943713/1048576 > >> > > > > > >> >> >> 09/08/12 09:58:32 INFO hbase.PerformanceEvaluation: > >> > client-2 > >> > > > > > >> >> >> 2097152/3040853/3145728 > >> > > > > > >> >> >> 09/08/12 09:59:14 INFO hbase.PerformanceEvaluation: > >> > client-1 > >> > > > > > >> >> >> 1048576/1992283/2097152 > >> > > > > > >> >> >> 09/08/12 10:02:28 INFO hbase.PerformanceEvaluation: > >> > client-0 > >> > > > > > >> >> >> 0/1048570/1048576 > >> > > > > > >> >> >> 09/08/12 10:02:30 INFO hbase.PerformanceEvaluation: > >> > client-0 > >> > > > > > >> Finished > >> > > > > > >> >> >> randomWrite in 2376615ms at offset 0 for 1048576 rows > >> > > > > > >> >> >> 09/08/12 10:02:30 INFO hbase.PerformanceEvaluation: > >> > Finished > >> > > 0 > >> > > > > in > >> > > > > > >> >> >> 2376615ms > >> > > > > > >> >> >> writing 1048576 rows > >> > > > > > >> >> >> 09/08/12 10:06:35 INFO hbase.PerformanceEvaluation: > >> > client-2 > >> > > > > > >> >> >> 2097152/3145710/3145728 > >> > > > > > >> >> >> 09/08/12 10:06:38 INFO hbase.PerformanceEvaluation: > >> > client-2 > >> > > > > > >> Finished > >> > > > > > >> >> >> randomWrite in 2623395ms at offset 2097152 for > 1048576 > >> > rows > >> > > > > > >> >> >> 09/08/12 10:06:38 INFO hbase.PerformanceEvaluation: > >> > Finished > >> > > 2 > >> > > > > in > >> > > > > > >> >> >> 2623395ms > >> > > > > > >> >> >> writing 1048576 rows > >> > > > > > >> >> >> 09/08/12 10:06:42 INFO hbase.PerformanceEvaluation: > >> > client-1 > >> > > > > > >> >> >> 1048576/2097140/2097152 > >> > > > > > >> >> >> 09/08/12 10:06:43 INFO hbase.PerformanceEvaluation: > >> > client-1 > >> > > > > > >> Finished > >> > > > > > >> >> >> randomWrite in 2630199ms at offset 1048576 for > 1048576 > >> > rows > >> > > > > > >> >> >> 09/08/12 10:06:43 INFO hbase.PerformanceEvaluation: > >> > Finished > >> > > 1 > >> > > > > in > >> > > > > > >> >> >> 2630199ms > >> > > > > > >> >> >> writing 1048576 rows > >> > > > > > >> >> >> > >> > > > > > >> >> >> > >> > > > > > >> >> >> > >> > > > > > >> >> >> Seems kind of slow for ~3M records. I have a 4 node > >> > cluster > >> > > > up > >> > > > > at > >> > > > > > >> the > >> > > > > > >> >> >> moment. HMaster & Namenode running on same box. > >> > > > > > >> >> >> -- > >> > > > > > >> >> >> View this message in context: > >> > > > > > >> >> >> > >> > > > > > >> >> > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > http://www.nabble.com/HBase-in-a-real-world-application-tp24920888p24940922.html > >> > > > > > >> >> >> Sent from the HBase User mailing list archive at > >> > Nabble.com. > >> > > > > > >> >> >> > >> > > > > > >> >> >> > >> > > > > > >> >> > > >> > > > > > >> >> > > >> > > > > > >> >> > >> > > > > > >> >> -- > >> > > > > > >> >> View this message in context: > >> > > > > > >> >> > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > http://www.nabble.com/HBase-in-a-real-world-application-tp24920888p24943406.html > >> > > > > > >> >> Sent from the HBase User mailing list archive at > >> Nabble.com. > >> > > > > > >> >> > >> > > > > > >> >> > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> -- > >> > > > > > >> View this message in context: > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > http://www.nabble.com/HBase-in-a-real-world-application-tp24920888p24955595.html > >> > > > > > >> Sent from the HBase User mailing list archive at > Nabble.com. > >> > > > > > >> > >> > > > > > >> > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
