Making some wild guesses here. If your IO system cannot keep up with the write load, eventually it has to block the writers. For a while your writes are buffered in the memstore(s) but at some point they need to be flushed to disk. Many small files will lead to pad read performance, so these smaller files are compacted into larger files. When the number of files for a region is too large, clients are blocked. Watch the compaction queue size and the number for storefiles to see if that is the case. If read performance is not critical, you can increase the number of a allowed files before clients are blocked, you can also increase the memstore multiplier, which temporarily allows a memstore to grow larger than the configured size. Only do this is your writes are bursty. None of this helps when you have a sustained write load that is higher than the what the aggregate IO capacity of your cluster can absorb.
The .META. with 10 rows is probably due to the fact that by default HBase prefetches at most 10 regions of a table. -- Lars ________________________________ From: "[email protected]" <[email protected]> To: [email protected] Cc: "[email protected]" <[email protected]> Sent: Thursday, November 22, 2012 4:49 AM Subject: HBASE Benchmarking Hi, I tried to write one million records into the HBASE cluster with 5 nodes (Hbase 0.94.2 on Hadoop 1.0.4) 1. Methide: sequentialWrite 2. From log, I found that the process had to sleep 3 times (total 4012ms) 3. It scanned .META for max=10rows Any idea why it got max=10 rows, will this parameter "max=10 rows" slowdown the process? and why it had to sleep a while? Thanks 12/11/22 20:36:30 DEBUG client.HConnectionManager$HConnectionImplementation: Removed TestTable,0000055872,1353587583943.b1541003edf820761ffe29683fe02df4. for tableName=TestTable from cache because of 0000765900 12/11/22 20:36:30 DEBUG client.HConnectionManager$HConnectionImplementation: Retry 1, sleep for 1003ms! 12/11/22 20:36:31 DEBUG client.MetaScanner: Scanning .META. starting at row=TestTable,0000055872,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2b5356d5 12/11/22 20:36:31 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for TestTable,0000055872,1353587583943.b1541003edf820761ffe29683fe02df4. is m147:60020 12/11/22 20:36:31 DEBUG client.HConnectionManager$HConnectionImplementation: Removed TestTable,0000055872,1353587583943.b1541003edf820761ffe29683fe02df4. for tableName=TestTable from cache because of 0000765900 12/11/22 20:36:31 DEBUG client.HConnectionManager$HConnectionImplementation: Retry 2, sleep for 1004ms! 12/11/22 20:36:32 DEBUG client.MetaScanner: Scanning .META. starting at row=TestTable,0000055872,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2b5356d5 12/11/22 20:36:32 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for TestTable,0000055872,1353587583943.b1541003edf820761ffe29683fe02df4. is m147:60020 12/11/22 20:36:33 DEBUG client.HConnectionManager$HConnectionImplementation: Removed TestTable,0000055872,1353587583943.b1541003edf820761ffe29683fe02df4. for tableName=TestTable from cache because of 0000765900 12/11/22 20:36:33 DEBUG client.HConnectionManager$HConnectionImplementation: Retry 3, sleep for 2005ms! 12/11/22 20:36:35 DEBUG client.MetaScanner: Scanning .META. starting at row=TestTable,0000192832,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2b5356d5 12/11/22 20:36:35 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for TestTable,0000192832,1353587767402.0770fb8c83b64620f29a1e186aa8addb. is m147:60020
