[
https://issues.apache.org/jira/browse/HDFS-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306107#comment-17306107
]
Ayush Saxena commented on HDFS-15907:
-------------------------------------
I think he was talking about to change {{ConcurrentHashMap}} as well and have
methods synchronized accessing them? From this :
{quote}But we should avoid using ConcurrentHashMap. It is known to have
performance issues and adds a lot of memory overhead. So whoever is using ACLs
heavily will have larger namespace requirements - very bad for large clusters.
Would prefer proper synchronization of the methods in ReferenceCountMap.
{quote}
I didn't follow the previous Jira, So, not very sure if that is addressed or
something not related, just give a check before you folks conclude. Nothing
blocking as such from my side
> Reduce Memory Overhead of AclFeature by avoiding AtomicInteger
> --------------------------------------------------------------
>
> Key: HDFS-15907
> URL: https://issues.apache.org/jira/browse/HDFS-15907
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Attachments: HDFS-15907.001.patch
>
>
> In HDFS-15792 we made some changes to the AclFeature and ReferenceCountedMap
> classes to address a rare bug when loading the FSImage in parallel.
> One change we made was to replace an int inside AclFeature with an
> AtomicInteger to avoid synchronising the methods in AclFeature.
> Discussing this change with [~weichiu], he pointed out that while the
> AclFeature cache is intended to reduce the count of AclFeature objects, on a
> large cluster, it is possible for there to be many millions of AclFeature
> objects.
> Previously, the int will have taken 4 bytes of heap.
> By moving to a AtomicInteger, we probably have an overhead of:
> 4 bytes (or 8 if the heap is over 32GB) for a reference to the atomic long
> object
> 12 byte overhead for the java object
> 4 bytes inside the atomic long to store an int.
>
> So the total heap overhead has gone from 4 bytes to 20 bytes just to use an
> AtomicInteger.
> Therefore I think it makes sense to remove the AtomicInteger and just
> synchronise the methods of AclFeature where the value is incremented /
> decremented / retrieved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
