[ https://issues.apache.org/jira/browse/YARN-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Misha Dmitriev updated YARN-7320: --------------------------------- Attachment: YARN-7320.02.patch Addressed checkstyle comments. > Duplicate LiteralByteStrings in > SystemCredentialsForAppsProto.credentialsForApp_ > -------------------------------------------------------------------------------- > > Key: YARN-7320 > URL: https://issues.apache.org/jira/browse/YARN-7320 > Project: Hadoop YARN > Issue Type: Improvement > Reporter: Misha Dmitriev > Assignee: Misha Dmitriev > Attachments: YARN-7320.01.patch, YARN-7320.02.patch > > > Using jxray (www.jxray.com) I've analyzed several heap dumps from YARN > Resource Manager running in a big cluster. The tool uncovered several sources > of memory waste. One problem, which results in wasting more than a quarter of > all memory, is a large number of duplicate {{LiteralByteString}} objects > coming from the following reference chain: > {code} > 1,011,810K (26.9%): byte[]: 5416705 / 100% dup arrays (22108 unique) > ↖com.google.protobuf.LiteralByteString.bytes > ↖org.apache.hadoop.yarn.proto.YarnServerCommonServiceProtos$.credentialsForApp_ > ↖{j.u.ArrayList} > ↖j.u.Collections$UnmodifiableRandomAccessList.c > ↖org.apache.hadoop.yarn.proto.YarnServerCommonServiceProtos$NodeHeartbeatResponseProto.systemCredentialsForApps_ > ↖org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatResponsePBImpl.proto > ↖org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl.latestNodeHeartBeatResponse > ↖org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode.rmNode > ... > {code} > That is, collectively reference chains that look as above hold in memory 5.4 > million {{LiteralByteString}} objects, but only ~22 thousand of these objects > are unique. Deduplicating these objects, e.g. using a Google Object Interner > instance, would save ~1GB of memory. > It looks like the main place where the above {{LiteralByteString}}s are > created and attached to the {{SystemCredentialsForAppsProto}} objects is in > {{NodeHeartbeatResponsePBImpl.java}}, method > {{addSystemCredentialsToProto()}}. Probably adding a call to an interner > there will fix the problem. wi -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org