[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709301#comment-14709301
]
Yi Liu commented on HDFS-8900:
------------------------------
{quote}
The max size of the packed byte[] is 32KB * 32 = 1MB...
But such big sequential memory allocation could cause full GC when heap is
fragmented.
{quote}
We can say this would never happen in real case. Typical the xattr value size
is very small (several bytes to tens of bytes, much smaller than 32KB, this is
also the design goal of xattr, for system/security namespace xattr used by HDFS
core, the size are very small today; if applications use user/trusted xattr to
store large size value, the memory of NN itself will have serious problem, for
1MB size xattrs of an INode as in your example, 1TB memory is required if there
is only 1MB number Inodes). In real case, even for HDFS transparent encryption
which uses xattr to store lots of information (see FileEncryptionInfo), the
xattr value size is only 48bytes.
Of course the applications may use the xattr wrongly and use it to store large
size (several KB to tens of KB) value, to prevent this, I will add a threshold
not to pack those large size xattrs and still store them separately in a list
as original. For small size xattr, we will pack them. The threshold can be
1KB, in a common situation, the list storing large size xattr is null, so only
4 bytes.
> Compact XAttrs to optimize memory footprint.
> --------------------------------------------
>
> Key: HDFS-8900
> URL: https://issues.apache.org/jira/browse/HDFS-8900
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Yi Liu
> Assignee: Yi Liu
> Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
> HDFS-8900.003.patch
>
>
> {code}
> private final ImmutableList<XAttr> xAttrs;
> {code}
> Currently we use above in XAttrFeature, it's not efficient from memory point
> of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
> and each object has memory alignment.
> We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
