Looks like you spend 1/6 of your time doing the gets, good to know. For autoflush=false, if you fit the 4-5KB in a single Put, then it won't help as 1 put = 1 rpc. Batching them together almost always improve performance. The default buffer size is 2MB btw.
LZO should give you another big boost, at least if your data can be compressed in any way. Also watch out for stuff that takes a lot of time in your code like instantiating lots of HTables (reuse same as much as you can inside a single thread), use finals, etc. I saw a good bunch of people shooting themselves in the foot by writing poorly performant code, crazy how running 800M times the same slowish thing ends up taking hours! J-D On Fri, May 28, 2010 at 4:11 PM, Jacob Isaac <ja...@ebrary.com> wrote: > Here is the summary of the runs > > puts (~4-5k per row) > regionsize #rows Total time (ms) > 1G 82282053*2 301943742 > 512M 82287593*2 313119378 > 256M 82246314*2 433200105 > > gets ((~4-5k per row) > regionsize #rows Total time (ms) > 1G 82427685 90116726 > 512M 82421943 94878466 > 256M 82395487 108160178 > > Note : for the 256m run the hbase.hregion.memstore.flush.size=64m > and for the other two runs the hbase.hregion.memstore.flush.size=96m > > Regarding disabling autoflush - since there are large number of writes(~4k > per row) happening we would > have hit the hbase.client.write.buffer size every few seconds. > > ~Jacob > > On Fri, May 28, 2010 at 1:36 PM, Jacob Isaac <ja...@ebrary.com> wrote: > >> Vidhya - This is using HBase API. >> >> J-D - I do have timing info for inserts and gets - Let me process the data >> and will post the results. >> >> ~Jacob. >> >> >> On Fri, May 28, 2010 at 1:16 PM, Vidhyashankar Venkataraman < >> vidhy...@yahoo-inc.com> wrote: >> >>> Jacob, >>> Just curious: Is your observed upload throughput that of bulk importing >>> or using the Hbase API? >>> Thanks >>> Vidhya >>> >>> On 5/28/10 1:13 PM, "Jacob Isaac" <ja...@ebrary.com> wrote: >>> >>> Hi J-D >>> >>> The run was done on a reformatted hdfs. >>> >>> Disabling WAL is not an option for us bcos this will be our normal mode of >>> operation and durability is important to us. >>> It was poor choice of words - 'upload' by me - it is more like >>> periodic/continous writes >>> >>> hbase.regionserver.maxlogs was 256 although >>> hbase.regionserver.hlog.blocksize was the default. >>> >>> Did not use compression. And autoflush is default (true) >>> >>> Each of the 20 node is running custom server program that's reading and >>> writing to HBase >>> Max of 6 write threads per node and 1 thread reading >>> Also wanted to point out that in the current tests we are writing to two >>> tables and reading from only one >>> >>> ~Jacob >>> >>> On Fri, May 28, 2010 at 12:42 PM, Jean-Daniel Cryans <jdcry...@apache.org >>> >wrote: >>> >>> > If the table was already created, changing hbase.hregion.max.filesize >>> > and hbase.hregion.memstore.flush.size won't be considered, those are >>> > the default values for new tables. You can set it in the shell too, >>> > see the "alter" command. >>> > >>> > Also, did you restart HBase? Did you push the configs to all nodes? >>> > Did you disable writing to the WAL? If not, because durability is >>> > still important to you but you want to upload as fast as you can, I >>> > would recommend changing this too: >>> > >>> > hbase.regionserver.hlog.blocksize 134217728 >>> > >>> > hbase.regionserver.maxlogs 128 >>> > >>> > I forgot you had quite largish values, so that must affect the log >>> > rolling a _lot_. >>> > >>> > Finally, did you LZOed the table? From experience, it will only do >>> > good http://wiki.apache.org/hadoop/UsingLzoCompression >>> > >>> > And finally (for real this time), how are you uploading to HBase? How >>> > many clients? Are you even using the write buffer? >>> > >>> > >>> http://hadoop.apache.org/hbase/docs/r0.20.4/api/org/apache/hadoop/hbase/client/HTable.html#setAutoFlush(boolean) >>> > >>> > J-D >>> > >>> > On Fri, May 28, 2010 at 12:28 PM, Jacob Isaac <ja...@ebrary.com> wrote: >>> > > Did a run yesterday, posted the relevant parameters below. >>> > > Did not see any difference in throughput or total run time (~9 hrs) >>> > > >>> > > I am consistently getting about 5k rows/sec, each row around ~4-5k >>> > > using a 17 node Hbase on 20 node HDFS cluster >>> > > >>> > > How does it compare?? Can I juice it more? >>> > > >>> > > ~Jacob >>> > > >>> > > >>> > > <property> >>> > > <name>hbase.regionserver.handler.count</name> >>> > > <value>60</value> >>> > > </property> >>> > > >>> > > <property> >>> > > <name>hbase.hregion.max.filesize</name> >>> > > <value>1073741824</value> >>> > > </property> >>> > > >>> > > <property> >>> > > <name>hbase.hregion.memstore.flush.size</name> >>> > > <value>100663296</value> >>> > > </property> >>> > > >>> > > <property> >>> > > <name>hbase.hstore.blockingStoreFiles</name> >>> > > <value>15</value> >>> > > </property> >>> > > >>> > > <property> >>> > > <name>hbase.hstore.compactionThreshold</name> >>> > > <value>4</value> >>> > > </property> >>> > > >>> > > <property> >>> > > <name>hbase.hregion.memstore.block.multiplier</name> >>> > > <value>8</value> >>> > > </property> >>> > > >>> > > >>> > > >>> > > On Fri, May 28, 2010 at 10:15 AM, Jean-Daniel Cryans < >>> > jdcry...@apache.org>wrote: >>> > > >>> > >> Like I said in my first email, it helps for random reading when lots >>> > >> of RAM is available to HBase. But it won't help the write throughput. >>> > >> >>> > >> J-D >>> > >> >>> > >> On Fri, May 28, 2010 at 10:12 AM, Vidhyashankar Venkataraman >>> > >> <vidhy...@yahoo-inc.com> wrote: >>> > >> > I am not sure if I understood this right, but does changing >>> > >> hfile.block.cache.size also help? >>> > >> > >>> > >> > >>> > >> > On 5/27/10 3:27 PM, "Jean-Daniel Cryans" <jdcry...@apache.org> >>> wrote: >>> > >> > >>> > >> > Well we do have a couple of other configs for high write >>> throughput: >>> > >> > >>> > >> > <property> >>> > >> > <name>hbase.hstore.blockingStoreFiles</name> >>> > >> > <value>15</value> >>> > >> > </property> >>> > >> > <property> >>> > >> > <name>hbase.hregion.memstore.block.multiplier</name> >>> > >> > <value>8</value> >>> > >> > </property> >>> > >> > <property> >>> > >> > <name>hbase.regionserver.handler.count</name> >>> > >> > <value>60</value> >>> > >> > </property> >>> > >> > <property> >>> > >> > <name>hbase.regions.percheckin</name> >>> > >> > <value>100</value> >>> > >> > </property> >>> > >> > >>> > >> > The last one is for restarts. Uploading very fast, you will more >>> > >> > likely hit all the upper limits (blocking store file and memstore) >>> and >>> > >> > this will lower your throughput. Those configs relax that. Also for >>> > >> > speedier uploads we disable writing to the WAL >>> > >> > >>> > >> >>> > >>> http://hadoop.apache.org/hbase/docs/r0.20.4/api/org/apache/hadoop/hbase/client/Put.html#setWriteToWAL(boolean) >>> > >> . >>> > >> > If the job fails or any machines fails you'll have to restart it or >>> > >> > figure the whole, and you absolutely need to force flushes when the >>> MR >>> > >> > is done. >>> > >> > >>> > >> > J-D >>> > >> > >>> > >> > On Thu, May 27, 2010 at 2:57 PM, Jacob Isaac <ja...@ebrary.com> >>> > wrote: >>> > >> >> Thanks J-D >>> > >> >> >>> > >> >> Currently we are trying to find/optimize our load/write times - >>> > although >>> > >> in >>> > >> >> prod we expect it to be 25/75 (writes/reads) ratio. >>> > >> >> We are using long table model with only one column - row-size is >>> > >> typically ~ >>> > >> >> 4-5k >>> > >> >> >>> > >> >> As to your suggestion on not using even 50% of disk space - I >>> agree >>> > and >>> > >> was >>> > >> >> planning to use only ~30-40% (1.5T of 4T) for HDFS >>> > >> >> and as I reported earlier >>> > >> >> 4000 regi...@256m per region(with 3 replications) on 20 nodes == >>> > 150G >>> > >> >> per/node == 10% utilization >>> > >> >> >>> > >> >> while using 1GB as maxfilesize did you have to adjust other params >>> > such >>> > >> >> as hbase.hstore.compactionThreshold and >>> > >> hbase.hregion.memstore.flush.size. >>> > >> >> There is an interesting observation by Jonathan Gray >>> > documented/reported >>> > >> in >>> > >> >> HBASE-2375 - >>> > >> >> wondering whether that issue gets compounded when using 1G as the >>> > >> >> hbase.hregion.max.filesize >>> > >> >> >>> > >> >> Thx >>> > >> >> Jacob >>> > >> >> >>> > >> >> >>> > >> > >>> > >> > >>> > >> >>> > > >>> > >>> >>> >> >
