[
https://issues.apache.org/jira/browse/MAPREDUCE-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827151#comment-15827151
]
Karthik Kambatla commented on MAPREDUCE-6829:
---------------------------------------------
Patch looks pretty good. Few comments:
# I see we are operating based on the _MAX suffix. What happens to user
counters that end in _MAX? Is there a way to skip them?
# We don't need to handle MIN. My recommendation would be to avoid adding it
until it is needed. I am okay with keeping them in if you insist, but would
like to see a test.
# Can we also add a test that uses MiniMRCluster so we have an end-to-end test
as well.
> Add peak memory usage counter for each task
> -------------------------------------------
>
> Key: MAPREDUCE-6829
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6829
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Yufei Gu
> Assignee: Miklos Szegedi
> Attachments: MAPREDUCE-6829.000.patch
>
>
> Each task has counters PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES, which
> are snapshots of memory usage of that task. They are not sufficient for users
> to understand peak memory usage by that task, e.g. in order to diagnose task
> failures, tune job parameters or change application design. This new feature
> will add two more counters for each task: PHYSICAL_MEMORY_BYTES_MAX and
> VIRTUAL_MEMORY_BYTES_MAX.
> This JIRA has the same feature from MAPREDUCE-4710. I file this new YARN
> JIRA since MAPREDUCE-4710 is pretty old one from MR 1.x era, it more or less
> assumes a branch-1 architecture, should be close at this point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
