[
https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205748#comment-13205748
]
Thomas Graves commented on MAPREDUCE-3843:
------------------------------------------
Anupam, if you don't mind could you change the rest of the commands in the
ClusterSetup to use hadoop-daemon.sh and yarn-daemon.sh appropriately?
I tested starting and stopping the job history server. Everything worked and
the log files (both out and log) and job summary file (mapred-jobsummary.log)
were all created properly using the log4j.properties config below. it was
started with the command: $HADOOP_COMMON_HOME/sbin/mr-jobhistory-daemon.sh
--config $HADOOP_CONF_DIR/ start historyserver.
I think we need to do some cleanup of the mr-jobhistory-daemon.sh and other
scripts since right now its intermixing HADOOP and YARN variables. I think the
log4j.properties template conf file in the tree needs to also be updated. But
I think that can be done in follow up jiras.
hadoop.mapreduce.jobsummary.logger=${hadoop.root.logger}
hadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log
mapred.jobsummary.logger=INFO,console
log4j.logger.org.apache.hadoop.mapreduce.jobhistory.JobSummary=${mapred.jobsummary.logger}
log4j.additivity.org.apache.hadoop.mapreduce.jobhistory.JobSummary=false
log4j.appender.JSA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.JSA.File=${hadoop.log.dir}/mapred-jobsummary.log
log4j.appender.JSA.layout=org.apache.log4j.PatternLayout
log4j.appender.JSA.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.JSA.DatePattern=.yyyy-MM-dd
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch,
> MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone
> missing on the
> RM host.
> The job summary log appears to be interspersed in
> yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not
> exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19
> /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64
> -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log
> -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop
> -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37
> /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str=
> -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir=
> -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA
> -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop
> -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not
> exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
