[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336383#comment-16336383
]
Yongjun Zhang edited comment on HDFS-12051 at 1/23/18 9:32 PM:
---------------------------------------------------------------
Thanks for your comments [~szetszwo]. I thought you had reviewed the patch.
Misha described his tests at:
https://issues.apache.org/jira/browse/HDFS-12051?focusedCommentId=16084471&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16084471
and
https://issues.apache.org/jira/browse/HDFS-12051?focusedCommentId=16329891&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16329891
I agree more testing is always a good thing.
I did another round of review and found one thing I did not point out earlier
(I'm sorry about that). The patch obsoleted the old config
{code}
public static final String DFS_NAMENODE_NAME_CACHE_THRESHOLD_KEY =
"dfs.namenode.name.cache.threshold";
public static final int DFS_NAMENODE_NAME_CACHE_THRESHOLD_DEFAULT = 10;
{code}
and introduced new config
{code}
public static final String DFS_NAMENODE_NAME_CACHE_SIZE_KEY =
"dfs.namenode.name.cache.size";
public static final int DFS_NAMENODE_NAME_CACHE_SIZE_DEFAULT = 4 * 1024 *
1024;
{code}
We should make this visible to the user saying that the former is deprecated
because of the implementation change, and the new one can be used to config the
new implementation.
Would you please address this [~mi...@cloudera.com]?
Since I'm the only one who has reviewed the patch, I'm inviting [~manojg] for a
review (thanks Manoj). While he is reviewing, other folks are welcome to review.
Thanks.
was (Author: yzhangal):
Thanks for your comments [~szetszwo]. I thought you had reviewed the patch.
Misha described his tests at:
https://issues.apache.org/jira/browse/HDFS-12051?focusedCommentId=16084471&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16084471
and
https://issues.apache.org/jira/browse/HDFS-12051?focusedCommentId=16329891&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16329891
I agree more testing is always a good thing.
I did another round of review and found one thing I did not point out earlier.
The patch obsoleted the old config
{code}
public static final String DFS_NAMENODE_NAME_CACHE_THRESHOLD_KEY =
"dfs.namenode.name.cache.threshold";
public static final int DFS_NAMENODE_NAME_CACHE_THRESHOLD_DEFAULT = 10;
{code}
and introduced new config
{code}
public static final String DFS_NAMENODE_NAME_CACHE_SIZE_KEY =
"dfs.namenode.name.cache.size";
public static final int DFS_NAMENODE_NAME_CACHE_SIZE_DEFAULT = 4 * 1024 *
1024;
{code}
We should make this visible to the user saying that the former is deprecated
because of the implementation change, and the new one can be used to config the
new implementation.
Would you please address this [~mi...@cloudera.com]?
Since I'm the only one who has reviewed the patch, I'm inviting [~manojg] for a
review (thanks Manoj). While he is reviewing, other folks are welcome to review.
Thanks.
> Reimplement NameCache in NameNode: Intern duplicate byte[] arrays (mainly
> those denoting file/directory names) to save memory
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Misha Dmitriev
> Assignee: Misha Dmitriev
> Priority: Major
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code:java}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
> ....
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> ....
> {code}
> There are several other places in NameNode code which also produce duplicate
> {{byte[]}} arrays.
> To eliminate this duplication and reclaim memory, we will need to write a
> small class similar to StringInterner, but designed specifically for byte[]
> arrays.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
