Vladimir Rodionov commented on HBASE-20045:

After investigation, we found the JVM garbage collector (GC) contributed a lot 
to the latency spikes. We defined a metric called GC stall percentage to 
measure the percentage of time a Cassandra server was doing stop-the-world GC 
(Young Gen GC) and could not serve client requests. Here’s another graph that 
shows the GC stall percentage on our production Cassandra servers. It was 1.25% 
during the lowest traffic time windows, and could be as high as 2.5% during 
peak hours.

The graph shows that a Cassandra server instance could spend 2.5% of runtime on 
garbage collections instead of serving client requests. The GC overhead 
obviously had a big impact on our P99 latency, so if we could lower the GC 
stall percentage, we would be able to reduce our P99 latency significantly.

Original is here:

Instagram engineers confirm what I said already:
*Java GC is the major source of p99 (and up)* 

> When running compaction, cache recent blocks.
> ---------------------------------------------
>                 Key: HBASE-20045
>                 URL: https://issues.apache.org/jira/browse/HBASE-20045
>             Project: HBase
>          Issue Type: New Feature
>          Components: BlockCache, Compaction
>    Affects Versions: 2.0.0-beta-1
>            Reporter: Jean-Marc Spaggiari
>            Priority: Major
> HBase already allows to cache blocks on flush. This is very useful for 
> usecases where most queries are against recent data. However, as soon as 
> their is a compaction, those blocks are evicted. It will be interesting to 
> have a table level parameter to say "When compacting, cache blocks less than 
> 24 hours old". That way, when running compaction, all blocks where some data 
> are less than 24h hold, will be automatically cached. 
> Very useful for table design where there is TS in the key but a long history 
> (Like a year of sensor data).

This message was sent by Atlassian JIRA

Reply via email to