[ https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094930#comment-16094930 ]
Vladimir Rodionov edited comment on HBASE-17819 at 7/20/17 4:33 PM: -------------------------------------------------------------------- {quote} BlockCacheKey --------------- String hfileName - Ref - 4 long offset - 8 BlockType blockType - Ref - 4 boolean isPrimaryReplicaBlock - 1 Total = 12 (Object) + 17 = 29 BucketEntry ------------ int offsetBase - 4 int length - 4 byte offset1 - 1 byte deserialiserIndex - 1 long accessCounter - 8 BlockPriority priority - Ref - 4 volatile boolean markedForEvict - 1 AtomicInteger refCount - 16 + 4 long cachedTime - 8 Total = 12 (Object) + 51 = 63 ConcurrentHashMap Map.Entry - 40 blocksByHFile ConcurrentSkipListSet Entry - 40 Total = 29 + 63 + 80 = 172 {quote} Just couple corrections on you math, guys # compressed OOP (obj ref = 4 bytes) works up to 30.5GB of heap size. Many users already have more than that # object's fields layout is slightly different: n-byte types are aligned on n-bytes boundaries, therefore if you have for example, boolean and long fields in the object, the object's size is going to be 16 (overhead) + 8 + 8 = 32 and not 16 + 1+ 8. You should take into account also that total object size is always multiple of 8, so if you get 42, then its actually - 48, because next object starts on a 8-byte boundary. You can shave some bytes by just rearranging fields in the object in size descending order: first go 8 byte type, followed by 4-byte types, 2-byte and 1-byte at the end was (Author: vrodionov): {quote} BlockCacheKey --------------- String hfileName - Ref - 4 long offset - 8 BlockType blockType - Ref - 4 boolean isPrimaryReplicaBlock - 1 Total = 12 (Object) + 17 = 29 BucketEntry ------------ int offsetBase - 4 int length - 4 byte offset1 - 1 byte deserialiserIndex - 1 long accessCounter - 8 BlockPriority priority - Ref - 4 volatile boolean markedForEvict - 1 AtomicInteger refCount - 16 + 4 long cachedTime - 8 Total = 12 (Object) + 51 = 63 ConcurrentHashMap Map.Entry - 40 blocksByHFile ConcurrentSkipListSet Entry - 40 Total = 29 + 63 + 80 = 172 {quote} Just couple corrections on you math, guys # compressed OOP (obj ref = 4 bytes) works up to 30.5GB of heap size. Many users already have more than that # object's fields layout is slightly different: n-byte types are aligned on n-bytes boundaries, therefore if you have for example, boolean and long fields of the object is going to be 16 (overhead) + 8 + 8 = 32 and not 16 + 1+ 8. You should take into account also that total object size is always multiple of 8, so if you get 42, then its actually - 48, because next object starts on a 8-byte boundary. You can shave some bytes by just rearranging fields in the object in size descending order: first go 8 byte type, followed by 4-byte types, 2-byte and 1-byte at the end > Reduce the heap overhead for BucketCache > ---------------------------------------- > > Key: HBASE-17819 > URL: https://issues.apache.org/jira/browse/HBASE-17819 > Project: HBase > Issue Type: Sub-task > Components: BucketCache > Reporter: Anoop Sam John > Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch > > > We keep Bucket entry map in BucketCache. Below is the math for heapSize for > the key , value into this map. > BlockCacheKey > --------------- > String hfileName - Ref - 4 > long offset - 8 > BlockType blockType - Ref - 4 > boolean isPrimaryReplicaBlock - 1 > Total = 12 (Object) + 17 = 29 > BucketEntry > ------------ > int offsetBase - 4 > int length - 4 > byte offset1 - 1 > byte deserialiserIndex - 1 > long accessCounter - 8 > BlockPriority priority - Ref - 4 > volatile boolean markedForEvict - 1 > AtomicInteger refCount - 16 + 4 > long cachedTime - 8 > Total = 12 (Object) + 51 = 63 > ConcurrentHashMap Map.Entry - 40 > blocksByHFile ConcurrentSkipListSet Entry - 40 > Total = 29 + 63 + 80 = 172 > For 10 million blocks we will end up having 1.6GB of heap size. > This jira aims to reduce this as much as possible -- This message was sent by Atlassian JIRA (v6.4.14#64029)