[
https://issues.apache.org/jira/browse/MAPREDUCE-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421842#comment-13421842
]
Miomir Boljanovic commented on MAPREDUCE-4229:
----------------------------------------------
Hi Todd, Thanks for suggesting Guava's Interner
With the Interner, we can canonicalize counter names without filling up PermGen
store, thus we don't need to use GSet<String>.
Judging from the issue description, there are a few counters where string
instances are created every time counter name or display name is queried. But
so far, I managed to identify only following one:
@InterfaceAudience.Private
public static class FSCounter extends AbstractCounter {
final String scheme;
final FileSystemCounter key;
private long value;
public FSCounter(String scheme, FileSystemCounter ref) {
this.scheme = scheme;
key = ref;
}
@Override
public String getName() {
return NAME_JOINER.join(scheme, key.name());
}
@Override
public String getDisplayName() {
return DISP_JOINER.join(scheme, localizeCounterName(key.name()));
}
Could perhaps point me to some of the remaining ones?
> Intern counter names in the JT
> ------------------------------
>
> Key: MAPREDUCE-4229
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4229
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 1.0.2
> Reporter: Todd Lipcon
>
> In our experience, most of the memory in production JTs goes to storing
> counter names (String objects and character arrays). Since most counter names
> are reused again and again, it would be a big memory savings to keep a hash
> set of already-used counter names within a job, and refer to the same object
> from all tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
