[
https://issues.apache.org/jira/browse/MAPREDUCE-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470981#comment-13470981
]
Todd Lipcon commented on MAPREDUCE-4469:
----------------------------------------
- the two overloads of getProcessOwner are repetitive - you could implement one
in terms of the other
- I'd like to avoid the shell command usages here - forks can be reasonably
expensive and also blow up the virtual size of the process tree. Instead, I
think it's sufficient to check something like:
{code}
boolean isPidOwnedByMe(int pid) {
File pidDir = new File(procFsDir, String.valueOf(pid));
// the environment of a process is only readable by the user
// running that process
File fileInPidDir = new File(pidDir, "environ");
return fileInPidDir.canRead();
}
{code}
This wouldn't cause extra forks and I think would have equivalent results, so
long as the user running the code isn't root.
----
- excludeList should be named something like {{excludedPids}}
- The following would be faster using StringUtils.split:
{code}
String[] strSplitted = str.split("\\s+");
{code}
(since the data is exactly space-delimited, we dont need the full overhead of a
regex)
- Rather than setting the configuration as a number of updates to skip, I think
it's better to set an interval at which the data will be re-calculated.
Otherwise the value of this configuration is scaled with respect to another
configuration (the counter update interval), which gets messy.
- I'm wondering, if we did the owner exclusion, and we improved the parsing to
use StringUtils.split, if it might be fast enough to avoid having to cache the
exclusions. Any chance you could run a benchmark? My concern is that the
exclusion idea doesn't work when pids get recycled (there are only 64k of them
so this happens pretty often)
> Resource calculation in child tasks is CPU-heavy
> ------------------------------------------------
>
> Key: MAPREDUCE-4469
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4469
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: performance, task
> Affects Versions: 1.0.3
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-4469.patch, MAPREDUCE-4469_rev2.patch,
> MAPREDUCE-4469_rev3.patch, MAPREDUCE-4469_rev4.patch
>
>
> In doing some benchmarking on a hadoop-1 derived codebase, I noticed that
> each of the child tasks was doing a ton of syscalls. Upon stracing, I noticed
> that it's spending a lot of time looping through all the files in /proc to
> calculate resource usage.
> As a test, I added a flag to disable use of the ResourceCalculatorPlugin
> within the tasks. On a CPU-bound 500G-sort workload, this improved total job
> runtime by about 10% (map slot-seconds by 14%, reduce slot seconds by 8%)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
