Hi Ilya,
In a normal case AM memory requirement should not be more than the default
for small sized jobs, but seems to be something erroneous in your case, Would
like to have more information :
1. Which version of Hadoop/ YARN ?
2. From the logs is it getting killed due to over usage of Vmem or Physical
memory ?
3. Can you run " jmap -histo -F <PID of AM container>" and share the heap dump
result?
4. If possible can you pastebin the AM logs?
+ Naga
________________________________________
From: Ilya Karpov [[email protected]]
Sent: Tuesday, September 22, 2015 21:06
To: [email protected]
Subject: Why would ApplicationManager request RAM more that defaut 1GB?
Hi all,
can’t figure out subj.
On my hadoop cluster I have an issue when ApplicationMaster(AM) killed by
NodeManager because AM tries to allocate more than default 1GB. MR application,
that AM is in charge of, is a mapper only job(1(!) mapper, no reducers,
downloads data from remote source). At the moment when AM killed, MR job is ok
(uses about 70% of ram limit). MR job doesn't have any custom counters,
distributes caches etc, just downloads data (by portions) via custom input
format. To fix this issue, I raised memory limit for AM, but I want to know
what is the reason of eating 1GB (!) for a trivial job like mine?
