Hi, Our HBase-backed app does about 95% reads and 5% writes on average, but once per hour we do a bulk update of several million rows (through HTable w/ large write buffer, not MR). Most of the time the bulk update has no impact on overall HBase performance. A few times during the day, usually during peak load times, the bulk update seems to result in a (minor?) compaction which has a huge impact on random read performance. Read throughput drops from ~4000 reads/sec down to ~1000 reads/sec. I'm wondering what strategies we can use to reduce the impact of compactions on random read performance. A couple ideas I had:
1. Do more frequent, smaller minor compactions. I guess we would accomplish this by lowering hbase.hstore.compactionThreshold, hbase.hstore.blockingStoreFiles, and/or hbase.hstore.compaction.max? 2. Try to prevent compactions altogether and just cron one major compaction per day when the system load is at its lowest. Not sure that this is a good idea. Does anyone currently do this? 3. I noticed that we're sometimes getting messages like "Too many hlogs: logs=33, maxlogs=32; forcing flush of 24 regions(s)". Should we disable the write-ahead log when doing bulk updates? I'm not entirely clear on the relationship between log flushing/rolling and minor/major compactions. As I understand it, a log flush will create HFiles, which might then trigger a minor compaction. Is that correct? Would disabling WAL help? 4. Hardware upgrade. We're running one 7200RPM SATA disk per datanode/regionserver now, so our I/O throughput probably isn't great. We will soon be testing a new hardware configuration with 2 SSDs per node. I'm sure this will help, but I'm looking for some short-term solutions that will work until we migrate to the new hardware. Have there been any performance improvements since 0.20.3 (other than HBASE-2180 which we already have) that might help? What is the best upgrade path if we were to upgrade our production HBase cluster is the next 1-2 weeks? 0.20.5? Build a snapshot from trunk/0.21? CDH3? Thanks, James
