[
https://issues.apache.org/jira/browse/HBASE-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886011#action_12886011
]
Jonathan Gray commented on HBASE-2821:
--------------------------------------
Sweet graph. Is there something you used to parse the NN log or something to
get this from? Would be quite interesting to look at this distribution and the
impact of HBASE-2375 and different compaction heuristics being considered.
> Keep young storefiles at lower replication
> ------------------------------------------
>
> Key: HBASE-2821
> URL: https://issues.apache.org/jira/browse/HBASE-2821
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Todd Lipcon
> Attachments: lifetime-distribution.png
>
>
> jgray and I were brainstorming some ideas about this:
> In a typical heavy-write scenario, many store files do not last very long.
> They're flushed and then within a small number of seconds a compaction runs
> and they get deleted. For these "short lifetime" store files, it's less
> likely that a failure will occur during the window in which they're valid.
> So, I think we can consider some optimizations like the following:
> - Flush files at replication count 2. Scan once a minute for any store files
> in the region that are older than 2 minutes. If they're found, increase their
> replication to 3. (alternatively, queue them to avoid scanning)
> - More dangerous: flush files at replication count 1, but don't count them
> when figuring log expiration. So, if they get lost, we force log splitting to
> recover.
> The performance gain here is that we avoid the network and disk transfer of
> writing the third replica for a file that we're just about to delete anyway.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
