You have yarn configured with log aggregation disabled ( yarn.log-aggregation-enable to false ). In this scenario, the logs are on the NodeManager where the AM was launched. For this, find the log-dirs property configured in yarn-site.xml and use that to find the logs for your application. If this is a multi-node cluster, you will need to find logs from all nodes in your cluster. YARN logs aggregation if enabled does this automatically and stores all logs in HDFS.
In any case, based on your earlier logs: > 2015-03-16 23:39:01,576 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClient - Tez Client Version: [ component=tez-api, > version=0.5.2, revision=5d728786b681e549a6f33375991dcce97cbf79f3, > SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, > buildTIme=20141104-1809 ] and > > org.apache.tez.client.TezClientUtils - Using tez.lib.uris > value from configuration: > hdfs://localhost:9000/apps/tez-0.6.0-SNAPSHOT/tez-0.6.0.tar.gz I think the inconsistency of versions might be the reason why things are failing. Please check the ResourceManager UI for the application in question and see if the diagnostics message points to a version issue. thanks — Hitesh On Mar 17, 2015, at 3:51 PM, dagriq <dag...@yahoo.ca> wrote: > hi > > and thanks for the quick reply. > I have tried the hint that you gave me but it was nothing written. > Please open the attached screenshot; I issed the command after error was > raised... > > > regards > > > -------------------------------------------- > On Tue, 3/17/15, Hitesh Shah <hit...@apache.org> wrote: > > Subject: Re: error after installation - TezSession has already shutdown > To: user@tez.apache.org > Received: Tuesday, March 17, 2015, 11:30 AM > > Hello > > First issue from the stack > trace: "org.apache.tez.dag.api.TezUncheckedException: > Invalid configuration of tez jars, tez.lib.uris is not > defined in the configuration”. It looks like your first > run failed due to this. The second run seemed to be > configured correctly but failed for a different reason. For > this, look for a line along the lines of > "org.apache.hadoop.yarn.client.api.impl.YarnClientImpl > - Submitted application application_1426563447689_0001” to > find the application ID. Now, invoke “bin/yarn logs > -applicationId <appId>” to get all the logs and > attach it to your email to this list so that we can take a > look to see if there are other issues. > > Actually, my guess is that you have 2 different > versions of tez in use. Based on the stack trace, you seem > to be using tez-0.5.2 on the client machine but your tarball > on HDFS is tez-0.6.0. Both of these need to have matching > versions. If you go to the ResourceManager and look at the > UI for the applicationId, its diagnostics might likely tell > you that the AM was incompatible with the client. If you see > a different error, please send across the logs from the > above command and we can help further. > > thanks > — HItesh > > > > > > > > > > On > Mar 17, 2015, at 6:27 AM, dagriq <dag...@yahoo.ca> > wrote: > >> hi >> >> I have downloaded > and compiled Tez and I received one error when I tried at > the bash shell: [ pig -x tez myscript.pig ]. >> >> I executed all steps > from the [ http://tez.apache.org/install.html ] and > I have a new folder in /usr/local/apache-tez. >> I added one more subfolder called conf > where I stored the new file tez-site.xml. >> My first error message is copied below: >> >> >> 2015-03-16 23:39:00,463 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler > - For vertex - scope-34: parallelism=1, memory=1024, java > opts=-Djava.net.preferIPv4Stack=true > -Dhadoop.metrics.log.level=WARN -Xmx200m > -Dlog4j.configuration=tez-container-log4j.properties > -Dyarn.app.container.log.dir=<LOG_DIR> > -Dtez.root.logger=INFO,CLA >> 2015-03-16 > 23:39:00,862 [PigTezLauncher-0] INFO > org.apache.pig.tools.pigstats.tez.TezScriptState - Pig > script settings are added to the job >> > 2015-03-16 23:39:01,576 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClient - Tez Client Version: [ > component=tez-api, version=0.5.2, > revision=5d728786b681e549a6f33375991dcce97cbf79f3, > SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, > buildTIme=20141104-1809 ] >> > 2015-03-16 23:39:01,757 [PigTezLauncher-0] INFO > org.apache.hadoop.yarn.client.RMProxy - Connecting to > ResourceManager at /0.0.0.0:8032 >> > 2015-03-16 23:39:02,798 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClient - Session mode. Starting > session. >> >> > 2015-03-16 23:39:02,800 [PigTezLauncher-0] ERROR > org.apache.pig.backend.hadoop.executionengine.tez.TezJob - > Cannot submit DAG >> >> > org.apache.tez.dag.api.TezUncheckedException: Invalid > configuration of tez jars, tez.lib.uris is not defined in > the configuration >> >> at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:162) >> at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:711) >> at > org.apache.tez.client.TezClient.start(TezClient.java:294) >> at > org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:95) >> at > org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:195) >> at > org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:159) >> at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:167) >> at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at > java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at > java.lang.Thread.run(Thread.java:745) >> > >> 2015-03-16 23:39:03,758 [main] INFO > org.apache.pig.tools.pigstats.tez.TezPigScriptStats - Script > Statistics: >> >> > HadoopVersion: 2.6.0 > > > >> > PigVersion: 0.14.0 > > >> TezVersion: 0.5.2 > > > >> > UserId: dagriq > > >> FileName: > dev2/email4.pig > > >> > StartedAt: 2015-03-16 23:38:55 > > > >> > FinishedAt: 2015-03-16 23:39:03 > > >> Features: FILTER > > > >> >> > Failed! >> >> DAG > PigLatin:dummy data-0_scope-0: >> > ApplicationId: null > > >> TotalLaunchedTasks: -1 > > > >> > FileBytesRead: -1 > > >> FileBytesWritten: -1 > > > >> > HdfsBytesRead: 0 > > > >> HdfsBytesWritten: > 0 > > >> > >> Input(s): >> >> Output(s): >> >> 2015-03-16 23:39:03,781 [main] ERROR > org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job > failed, hadoop does not return any error message >> Details at logfile: > /usr/local/pig/logs/pig-err.log >> > 2015-03-16 23:39:03,860 [main] INFO org.apache.pig.Main - > Pig script completed in 19 seconds and 936 milliseconds > (19936 ms) >> 2015-03-16 23:39:03,860 > [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher > - Shutting down thread pool >> >> So I copied the tez-site.xml to the hadoop > files and I re-executed the above mentioned command: pig -x > tez myscript.pig. I received another error , I am copying it > below: >> >> 2015-03-16 > 23:40:28,679 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezCompiler > - File concatenation threshold: 100 optimistic? false >> 2015-03-16 23:40:28,805 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapreduce.inputformat.class is deprecated. Instead, use > mapreduce.job.inputformat.class >> > 2015-03-16 23:40:29,050 [main] INFO > org.apache.tez.mapreduce.hadoop.MRInputHelpers - Generating > mapreduce api input splits >> 2015-03-16 > 23:40:29,276 [main] INFO > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - > Total input paths to process : 4 >> > 2015-03-16 23:40:29,280 [main] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil > - Total input paths to process : 4 >> > 2015-03-16 23:40:29,449 [main] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil > - Total input paths (combined) to process : 1 >> 2015-03-16 23:40:29,602 [main] INFO > org.apache.tez.mapreduce.hadoop.MRInputHelpers - NumSplits: > 1, SerializedSize: 614 >> 2015-03-16 > 23:40:31,773 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler > - Local resource: joda-time-2.1.jar >> > 2015-03-16 23:40:31,773 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler > - Local resource: guava-11.0.2.jar >> > 2015-03-16 23:40:31,774 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler > - Local resource: pig-0.14.0-core-h2.jar >> 2015-03-16 23:40:31,774 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler > - Local resource: antlr-runtime-3.4.jar >> > 2015-03-16 23:40:31,775 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler > - Local resource: automaton-1.11-8.jar >> > 2015-03-16 23:40:31,884 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > fs.default.name is deprecated. Instead, use fs.defaultFS >> 2015-03-16 23:40:32,359 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapred.output.compress is deprecated. Instead, use > mapreduce.output.fileoutputformat.compress >> 2015-03-16 23:40:32,365 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.shuffle.merge.percent to 0.66 from MR > setting mapreduce.reduce.shuffle.merge.percent >> 2015-03-16 23:40:32,366 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.shuffle.fetch.buffer.percent to 0.70 > from MR setting > mapreduce.reduce.shuffle.input.buffer.percent >> 2015-03-16 23:40:32,367 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.io.sort.mb to 100 from MR setting > mapreduce.task.io.sort.mb >> 2015-03-16 > 23:40:32,367 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.shuffle.memory.limit.percent to 0.25 > from MR setting > mapreduce.reduce.shuffle.memory.limit.percent >> 2015-03-16 23:40:32,368 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.io.sort.factor to 10 from MR setting > mapreduce.task.io.sort.factor >> > 2015-03-16 23:40:32,368 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.shuffle.connect.timeout to 180000 from > MR setting mapreduce.reduce.shuffle.connect.timeout >> 2015-03-16 23:40:32,369 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.internal.sorter.class to > org.apache.hadoop.util.QuickSort from MR setting > map.sort.class >> 2015-03-16 23:40:32,370 > [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.merge.progress.records to 10000 from > MR setting mapreduce.task.merge.progress.records >> 2015-03-16 23:40:32,370 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.compress to false from MR setting > mapreduce.map.output.compress >> > 2015-03-16 23:40:32,371 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.sort.spill.percent to 0.80 from MR > setting mapreduce.map.sort.spill.percent >> 2015-03-16 23:40:32,372 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.shuffle.ssl.enable to false from MR > setting mapreduce.shuffle.ssl.enabled >> > 2015-03-16 23:40:32,373 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.ifile.readahead to true from MR > setting mapreduce.ifile.readahead >> > 2015-03-16 23:40:32,373 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.shuffle.parallel.copies to 5 from MR > setting mapreduce.reduce.shuffle.parallelcopies >> 2015-03-16 23:40:32,374 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.ifile.readahead.bytes to 4194304 from > MR setting mapreduce.ifile.readahead.bytes >> 2015-03-16 23:40:32,374 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.task.input.post-merge.buffer.percent > to 0.0 from MR setting > mapreduce.reduce.input.buffer.percent >> > 2015-03-16 23:40:32,375 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.shuffle.read.timeout to 180000 from MR > setting mapreduce.reduce.shuffle.read.timeout >> 2015-03-16 23:40:32,375 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper > - Setting tez.runtime.compress.codec to > org.apache.hadoop.io.compress.DefaultCodec from MR setting > mapreduce.map.output.compress.codec >> > 2015-03-16 23:40:32,707 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler > - For vertex - scope-34: parallelism=1, memory=1024, java > opts=-Djava.net.preferIPv4Stack=true > -Dhadoop.metrics.log.level=WARN -Xmx200m > -Dlog4j.configuration=tez-container-log4j.properties > -Dyarn.app.container.log.dir=<LOG_DIR> > -Dtez.root.logger=INFO,CLA >> 2015-03-16 > 23:40:33,187 [PigTezLauncher-0] INFO > org.apache.pig.tools.pigstats.tez.TezScriptState - Pig > script settings are added to the job >> > 2015-03-16 23:40:33,899 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClient - Tez Client Version: [ > component=tez-api, version=0.5.2, > revision=5d728786b681e549a6f33375991dcce97cbf79f3, > SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, > buildTIme=20141104-1809 ] >> > 2015-03-16 23:40:34,098 [PigTezLauncher-0] INFO > org.apache.hadoop.yarn.client.RMProxy - Connecting to > ResourceManager at /0.0.0.0:8032 >> > 2015-03-16 23:40:34,923 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClient - Session mode. Starting > session. >> 2015-03-16 23:40:34,923 > [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - > Session mode. Starting session. >> > 2015-03-16 23:40:34,924 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClientUtils - Using tez.lib.uris > value from configuration: > hdfs://localhost:9000/apps/tez-0.6.0-SNAPSHOT/tez-0.6.0.tar.gz >> 2015-03-16 23:40:35,129 [PigTezLauncher-0] > INFO org.apache.tez.client.TezClient - Tez system stage > directory > hdfs://localhost:9000/tmp/temp-1303134320/.tez/application_1426563447689_0001 > doesn't exist and is created >> > 2015-03-16 23:40:36,048 [PigTezLauncher-0] INFO > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - > Submitted application application_1426563447689_0001 >> 2015-03-16 23:40:36,055 [PigTezLauncher-0] > INFO org.apache.tez.client.TezClient - The url to track > the Tez Session: http://optimf:8088/proxy/application_1426563447689_0001/ >> >> 2015-03-16 > 23:40:59,709 [PigTezLauncher-0] ERROR > org.apache.pig.backend.hadoop.executionengine.tez.TezJob - > Cannot submit DAG >> > org.apache.tez.dag.api.SessionNotRunning: TezSession has > already shutdown >> at > org.apache.tez.client.TezClient.waitTillReady(TezClient.java:603) >> at > org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:100) >> at > org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:195) >> at > org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:159) >> at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:167) >> at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at > java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at > java.lang.Thread.run(Thread.java:745) >> > 2015-03-16 23:41:00,014 [main] INFO > org.apache.pig.tools.pigstats.tez.TezPigScriptStats - Script > Statistics: >> >> > HadoopVersion: 2.6.0 > > > >> > PigVersion: 0.14.0 > > >> TezVersion: 0.5.2 > > > >> > UserId: dagriq > > >> FileName: > dev2/email4.pig > > >> > StartedAt: 2015-03-16 23:40:28 > > > >> > FinishedAt: 2015-03-16 23:41:00 > > >> Features: FILTER > > > >> >> > Failed! >> >> >> DAG PigLatin:dummy data-0_scope-0: >> ApplicationId: null > > > >> > TotalLaunchedTasks: -1 > > >> FileBytesRead: -1 > > > >> > FileBytesWritten: -1 > > >> HdfsBytesRead: 0 > > > >> > HdfsBytesWritten: 0 > > >> >> Input(s): >> >> Output(s): >> >> 2015-03-16 > 23:41:00,036 [main] ERROR > org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job > failed, hadoop does not return any error message >> Details at logfile: > /usr/local/pig/logs/pig-err.log >> > 2015-03-16 23:41:00,146 [main] INFO org.apache.pig.Main - > Pig script completed in 42 seconds and 897 milliseconds > (42897 ms) >> 2015-03-16 23:41:00,147 > [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher > - Shutting down thread pool >> >> >> What is wrong in my > installation ? >> Thank you for your > help! >> > <2015-03-17-184721_1280x1024_scrot.png>
