Stephen O'Donnell created HDFS-15907:
----------------------------------------
Summary: Reduce Memory Overhead of AclFeature by avoiding
AtomicInteger
Key: HDFS-15907
URL: https://issues.apache.org/jira/browse/HDFS-15907
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
In HDFS-15792 we made some changes to the AclFeature and ReferenceCountedMap
classes to address a rare bug when loading the FSImage in parallel.
One change we made was to replace an int inside AclFeature with an
AtomicInteger to avoid synchronising the methods in AclFeature.
Discussing this change with [~weichiu], he pointed out that while the
AclFeature cache is intended to reduce the count of AclFeature objects, on a
large cluster, it is possible for there to be many millions of AclFeature
objects.
Previously, the int will have taken 4 bytes of heap.
By moving to a AtomicInteger, we probably have an overhead of:
4 bytes (or 8 if the heap is over 32GB) for a reference to the atomic long
object
12 byte overhead for the java object
4 bytes inside the atomic long to store an int.
So the total heap overhead has gone from 4 bytes to 20 bytes just to use an
AtomicInteger.
Therefore I think it makes sense to remove the AtomicInteger and just
synchronise the methods of AclFeature where the value is incremented /
decremented / retrieved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
