[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709050#comment-14709050
]
Brahma Reddy Battula commented on HDFS-8900:
--------------------------------------------
Thanks [~luyi0619] for working on this...
I have a little concern about GC problem. since now {{Xattr.value}} is a byte[].
In the old design, When we update a Xattr, we doesn't changes other xattrs. In
the proposed design, the values of all xattrs are combined in one packed byte[]
.
The packed byte[] could be large. Everytime we change one xattr, we have to
dispose the packed byte[] and create a new one, even though other xattrs are
not updated.It may be not a problem when set an xattr for a file. But when we
recursively set for a directory, It burdens the GC. The max size of the packed
byte[] is 32KB * 32 = 1MB. It's unlikely to reach that. But such big sequential
memory allocation could cause full GC when heap is fragmented.
So, maybe we could use byte array or use off-heap memory(may slow-down
operations..?) for Xattrs.
> Compact XAttrs to optimize memory footprint.
> --------------------------------------------
>
> Key: HDFS-8900
> URL: https://issues.apache.org/jira/browse/HDFS-8900
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Yi Liu
> Assignee: Yi Liu
> Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
> HDFS-8900.003.patch
>
>
> {code}
> private final ImmutableList<XAttr> xAttrs;
> {code}
> Currently we use above in XAttrFeature, it's not efficient from memory point
> of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
> and each object has memory alignment.
> We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
