Hi Oussama, thanks for your suggestions!
Via yarn logs we could also not get the logs. But we just solved the issue. It was an issue with Kerberos. After setting mapreduce.jobtracker.kerberos.principal dfs.namenode.kerberos.principal the logs from the Mappers and Reducers are in the yarn application logs. Best regards, Tobias -----Ursprüngliche Nachricht----- Von: Oussama Chougna [mailto:[email protected]] Gesendet: Mittwoch, 9. September 2015 15:51 An: [email protected] Betreff: RE: AW: AW: Logging using Log4J in MapReduce action Hi Tobias, Ok, write down the job id of the failing job and try the following: Open a terminal shell to one of your hadoop nodes and issue the following command: yarn logs -applicationId <job id> substitute 'job id' with the id of the MR job. This will print out all log information that is available. (App Master logs, mapper logs, reducers logs...etc) Oussama Chougna > From: [email protected] > To: [email protected] > Subject: AW: AW: Logging using Log4J in MapReduce action > Date: Wed, 9 Sep 2015 12:24:13 +0000 > > Hi Oussama, > > this is exactly my problem: when I do run the MR job inside a MR in Oozie, > the logs from the Mappers and Reducers do not appear in the logs on the Job > History Server. > > Best, > > Tobias > > -----Ursprüngliche Nachricht----- > Von: Oussama Chougna [mailto:[email protected]] > Gesendet: Mittwoch, 9. September 2015 14:17 > An: [email protected] > Betreff: RE: AW: Logging using Log4J in MapReduce action > > Hi Tobias, > Oozie will submit your MR job to YARN.Check the job history server for the > job logs, it is accessible via a browser on http://<your host>:19888. (19888 > is the default port).In this UI click on the failed job, then you can consult > the logs of the mappers and reducers. > Good luck, > > Oussama Chougna > > > From: [email protected] > > To: [email protected] > > Subject: AW: Logging using Log4J in MapReduce action > > Date: Wed, 9 Sep 2015 12:01:44 +0000 > > > > Hi Oussama, > > > > we are using YARN. And we do have a kerberized cluster. I do submit jobs > > via the Command Line like > > > > oozie job -config jobtr1vi jo.properties -run > > > > In my job properties I do set the following properties: > > > > oozie.use.system.libpath=true > > security_enabled=true > > nameNode=hdfs://<mynamenodedns>:8020 > > jobTracker=<jobtrackerdns>:8032 > > oozie.wf.application.path=hdfs://<path-to-workflowxml>/workflow.xml > > > > Best, > > > > Tobias > > > > -----Ursprüngliche Nachricht----- > > Von: Oussama Chougna [mailto:[email protected]] > > Gesendet: Mittwoch, 9. September 2015 13:57 > > An: [email protected] > > Betreff: RE: Logging using Log4J in MapReduce action > > > > Hi Tobias, > > Are you using MRv1 or YARN? How do you submit the jobs? Command line? > > > > Cheers, > > > > Oussama Chougna > > > > > From: [email protected] > > > To: [email protected] > > > Subject: Logging using Log4J in MapReduce action > > > Date: Wed, 9 Sep 2015 08:12:04 +0000 > > > > > > Hello everyone, > > > > > > we did implement a couple of custom MapReduce jobs which we plan to > > > schedule using Oozie workflows in MapReduce actions. > > > > > > All MR jobs are running fine outside of Oozie, but one of them is not. > > > > > > The problem we have is that if those MR jobs are run inside Oozie nothing > > > is get logged using the Log4J loggers that we are registering within the > > > Mapper and Reducer classes using e.g. "private static final Logger logger > > > = Logger.getLogger(TransformMapper.class);" > > > The logging works fine if the MR jobs are run outside of Oozie and > > > logging messages appear in the task attempt logs, but not when they are > > > run within Oozie. > > > > > > Does anyone face similar issues? > > > > > > Does anyone know a solution to this problem? > > > > > > Maybe it is just a matter of configuring something somewhere > > > appropriately. > > > > > > Thanks in advance for your answers, > > > > > > Tobias > > > > > > -- > > > Tobias B. > > > > > >
