[
https://issues.apache.org/jira/browse/HBASE-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726754#action_12726754
]
Jonathan Gray commented on HBASE-1607:
--------------------------------------
Did I comment out stuff? Didn't think I had. This patch might remove
previously commented out stuff.
ClassSize is a utility to help us size classes. It generates the sizes for the
different native java classes that we use in heap sizable classes. It does
nothing with an instantiated class.
I agree that FIXED_OVERHEAD and DEEP_OVERHEAD are not necessarily the best long
term solution out there :) Currently in trunk, our MemStore is sized with this
beautiful line:
{noformat}
private final static int ESTIMATED_KV_HEAP_TAX = 60;
long heapSize(final KeyValue kv, final boolean notpresent) {
return notpresent?
// Add overhead for value byte array and for Map.Entry -- 57 bytes
// on x64 according to jprofiler.
ESTIMATED_KV_HEAP_TAX + 57 + kv.getLength(): 0; // Guess no change in
size.
}
{noformat}
So this is a vast improvement. It might look nasty but it's not a hard-coded
number, rather it is calculated. Interestingly, 57 bytes is not a valid size
for something in memory because everything is aligned (4 byte on 32bit, 8 byte
on 64bit).
Again, I set out with two goals. Make sizing as accurate as possible, and make
tests so that if any of our sized classes change the tests will fail. This has
been done with this patch and those before it.
So FIXED_OVERHEAD might look confusing, but it's principled and shows where the
sizing is coming from. When the unit test fails, ClassSize outputs it's sizing
in debug mode so you can very easily see exactly where you missed. Message is
something like: Expected <104> but got <96>. And there is output of how many
references, primitives, etc it found. Quite simple to fix. If you want to
play with it, just modify one of the FIXED_OVERHEADs and run TestHeapSize.
This is not a permanent solution, perhaps, but works well and is far better
than the hard coded values we have now.
> Redo MemStore heap sizing to be accurate, testable, and more like new
> LruBlockCache
> -----------------------------------------------------------------------------------
>
> Key: HBASE-1607
> URL: https://issues.apache.org/jira/browse/HBASE-1607
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1607-v1.patch
>
>
> MemStore sizing is inaccurate and does not include all overhead.
> I'm going to make it look like the LruBlockCache does. Will provide a
> MemStore.heapSize() method that includes ALL overhead of the MemStore itself.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
