Esteban Gutierrez created HBASE-13407:
-----------------------------------------
Summary: Add a configurable jitter to MemStoreFlusher#FlushHandler
in order to smooth write latency
Key: HBASE-13407
URL: https://issues.apache.org/jira/browse/HBASE-13407
Project: HBase
Issue Type: Improvement
Reporter: Esteban Gutierrez
There is a very interesting behavior that I can reproduce consistently with
many workloads from HBase 0.98 to HBase 1.0 since hbase.hstore.flusher.count
was set by default to 2: when writes are evenly distributed across regions,
memstores grow and flush about the same rate causing spikes in IO and CPU. The
side effect of those spikes is loss in throughput which in some cases can above
10% impacting write metrics. When the flushes get a out of sync the spikes
lower and and throughput is very stable. Reverting hbase.hstore.flusher.count
to 1 doesn't help too much with write heavy workloads since we end with a large
flush queue that eventually can block writers.
Adding a small configurable jitter hbase.server.thread.wakefrequency.jitter.pct
(a percentage of the hbase.server.thread.wakefrequency frequency) can help to
stagger the writes from FlushHandler to HDFS and smooth the write latencies
when the memstores are flushed in multiple threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
