Hi Harsh, I created the following JIRA for NodeManager not reporting appropriate errors when NM Memory < AM Memory: MAPREDUCE-4508 - YARN needs to properly check the NM,AM memory properties in yarn-site.xml and mapred.xml and report errors accordingly.<https://issues.apache.org/jira/browse/MAPREDUCE-4508>
Please let me know if anything else if required for the JIRA. Thanks, Anil Gupta On Tue, Jul 31, 2012 at 11:26 AM, anil gupta <anilgupt...@gmail.com> wrote: > Hi Harsh and Others, > > I was able to run the job when I login as user "hdfs". However, it fails > if i run it as "root". I was suspecting this as a problem before also and > it came out to be true. > > Thanks, > Anil gupta > > > On Mon, Jul 30, 2012 at 9:21 PM, abhiTowson cal <abhishek.dod...@gmail.com > > wrote: > >> Hi anil, >> >> Was trying several things.I didn't hadoop-env.sh, so i created it. >> >> Regards >> Abhishek >> >> On Mon, Jul 30, 2012 at 11:51 PM, anil gupta <anilgupt...@gmail.com> >> wrote: >> > That's pretty interesting!! From where you figured out that you need to >> add >> > that property? Just trying to understand how adding that property fixed >> the >> > issue. >> > >> > On Mon, Jul 30, 2012 at 8:12 PM, abhiTowson cal >> > <abhishek.dod...@gmail.com>wrote: >> > >> >> Hi anil, >> >> >> >> Adding property resolved issue for me, and i also made this change >> >> >> >> vim hadoop-env.sh >> >> >> >> export JAVA_HOME=/usr/lib/java-1.6.0/jdk1.6.0_33 >> >> if [ "$JAVA_HOME" != "" ]; then >> >> #echo "run java in $JAVA_HOME" >> >> JAVA_HOME=$JAVA_HOME >> >> fi >> >> >> >> if [ "$JAVA_HOME" = "" ]; then >> >> echo "Error: JAVA_HOME is not set." >> >> exit 1 >> >> fi >> >> >> >> JAVA=$JAVA_HOME/bin/java >> >> JAVA_HEAP_MAX=-Xmx1000m >> >> >> >> Regards >> >> Abhishek >> >> >> >> >> >> On Mon, Jul 30, 2012 at 10:47 PM, anil gupta <anilgupt...@gmail.com> >> >> wrote: >> >> > Hi Abhishek, >> >> > >> >> > Did you mean that adding >> yarn.resourcemanager.resource-tracker.address >> >> > along with yarn.log-aggregation-enable in my configuration will >> resolve >> >> the >> >> > problem in which map-reduce job fails at 0% with the following >> error: In >> >> > the web page of >> >> > >> >> >> http://data-node:8042/node/containerlogs/container_1343687008058_0003_01_000001/rootthe >> >> > page says: >> >> > Failed redirect for container_1343687008058_0003_01_000001 Failed >> while >> >> > trying to construct the redirect url to the log server. Log Server >> url >> >> may >> >> > not be configured. Unknown container. Container either has not >> started or >> >> > has already completed or doesn't belong to this node at all. >> >> > Please let me know. >> >> > >> >> > Thanks, >> >> > Anil Gupta >> >> > >> >> > On Mon, Jul 30, 2012 at 7:30 PM, abhiTowson cal >> >> > <abhishek.dod...@gmail.com>wrote: >> >> > >> >> >> hi anil, >> >> >> >> >> >> Adding these help me resolve the issue for me >> >> >> yarn.resourcemanager.resource-tracker.address >> >> >> >> >> >> Regards >> >> >> Abhishek >> >> >> >> >> >> On Mon, Jul 30, 2012 at 7:56 PM, anil gupta <anilgupt...@gmail.com> >> >> wrote: >> >> >> > Hi Rahul, >> >> >> > >> >> >> > Thanks for your response. I can certainly enable the >> >> >> > yarn.log-aggregation-enable to true. But after enabling this what >> >> manual >> >> >> > steps i will have to take to run jobs. Could you please elaborate. >> >> >> > >> >> >> > Thanks, >> >> >> > Anil >> >> >> > >> >> >> > On Mon, Jul 30, 2012 at 4:26 PM, Rahul Jain <rja...@gmail.com> >> wrote: >> >> >> > >> >> >> >> The inability to look at map-reduce logs for failed logs is due >> to >> >> >> number >> >> >> >> of open issues in yarn; see my recent comment here: >> >> >> >> >> >> >> >> >> >> >> >> >> >> https://issues.apache.org/jira/browse/MAPREDUCE-4428?focusedCommentId=13412995&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13412995 >> >> >> >> >> >> >> >> You can workaround this by enabling log aggregation and manually >> >> copying >> >> >> >> job logs from HDFS log location. Of course that is a painful way >> till >> >> >> the >> >> >> >> yarn log collection and history bugs are resolved in an upcoming >> >> >> release. >> >> >> >> >> >> >> >> -Rahul >> >> >> >> >> >> >> >> >> >> >> >> > 12/07/27 09:38:27 INFO mapred.ResourceMgrDelegate: Submitted >> >> >> application >> >> >> >> > application_1343365114818_0002 to ResourceManager at >> ihub-an-l1/ >> >> >> >> > 172.31.192.151:8040 >> >> >> >> > 12/07/27 09:38:27 INFO mapreduce.Job: The url to track the job: >> >> >> >> > http://ihub-an-l1:9999/proxy/application_1343365114818_0002/ >> >> >> >> > 12/07/27 09:38:27 INFO mapreduce.Job: Running job: >> >> >> job_1343365114818_0002 >> >> >> >> > >> >> >> >> > No Map-Reduce task are started by the cluster. I dont see any >> >> errors >> >> >> >> > anywhere in the application. Please help me in resolving this >> >> problem. >> >> >> >> > >> >> >> >> > Thanks, >> >> >> >> > Anil Gupta >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > Thanks & Regards, >> >> >> > Anil Gupta >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > Thanks & Regards, >> >> > Anil Gupta >> >> >> > >> > >> > >> > -- >> > Thanks & Regards, >> > Anil Gupta >> > > > > -- > Thanks & Regards, > Anil Gupta > -- Thanks & Regards, Anil Gupta
