hi I have downloaded and compiled Tez and I received one error when I tried at the bash shell: [ pig -x tez myscript.pig ].
I executed all steps from the [ http://tez.apache.org/install.html ] and I have a new folder in /usr/local/apache-tez. I added one more subfolder called conf where I stored the new file tez-site.xml. My first error message is copied below: 2015-03-16 23:39:00,463 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - For vertex - scope-34: parallelism=1, memory=1024, java opts=-Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA 2015-03-16 23:39:00,862 [PigTezLauncher-0] INFO org.apache.pig.tools.pigstats.tez.TezScriptState - Pig script settings are added to the job 2015-03-16 23:39:01,576 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Tez Client Version: [ component=tez-api, version=0.5.2, revision=5d728786b681e549a6f33375991dcce97cbf79f3, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTIme=20141104-1809 ] 2015-03-16 23:39:01,757 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8032 2015-03-16 23:39:02,798 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Session mode. Starting session. 2015-03-16 23:39:02,800 [PigTezLauncher-0] ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Cannot submit DAG org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configuration at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:162) at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:711) at org.apache.tez.client.TezClient.start(TezClient.java:294) at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:95) at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:195) at org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:159) at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:167) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-03-16 23:39:03,758 [main] INFO org.apache.pig.tools.pigstats.tez.TezPigScriptStats - Script Statistics: HadoopVersion: 2.6.0 PigVersion: 0.14.0 TezVersion: 0.5.2 UserId: dagriq FileName: dev2/email4.pig StartedAt: 2015-03-16 23:38:55 FinishedAt: 2015-03-16 23:39:03 Features: FILTER Failed! DAG PigLatin:dummy data-0_scope-0: ApplicationId: null TotalLaunchedTasks: -1 FileBytesRead: -1 FileBytesWritten: -1 HdfsBytesRead: 0 HdfsBytesWritten: 0 Input(s): Output(s): 2015-03-16 23:39:03,781 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message Details at logfile: /usr/local/pig/logs/pig-err.log 2015-03-16 23:39:03,860 [main] INFO org.apache.pig.Main - Pig script completed in 19 seconds and 936 milliseconds (19936 ms) 2015-03-16 23:39:03,860 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting down thread pool So I copied the tez-site.xml to the hadoop files and I re-executed the above mentioned command: pig -x tez myscript.pig. I received another error , I am copying it below: 2015-03-16 23:40:28,679 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.plan.TezCompiler - File concatenation threshold: 100 optimistic? false 2015-03-16 23:40:28,805 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class 2015-03-16 23:40:29,050 [main] INFO org.apache.tez.mapreduce.hadoop.MRInputHelpers - Generating mapreduce api input splits 2015-03-16 23:40:29,276 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 4 2015-03-16 23:40:29,280 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 4 2015-03-16 23:40:29,449 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 2015-03-16 23:40:29,602 [main] INFO org.apache.tez.mapreduce.hadoop.MRInputHelpers - NumSplits: 1, SerializedSize: 614 2015-03-16 23:40:31,773 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: joda-time-2.1.jar 2015-03-16 23:40:31,773 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: guava-11.0.2.jar 2015-03-16 23:40:31,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: pig-0.14.0-core-h2.jar 2015-03-16 23:40:31,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: antlr-runtime-3.4.jar 2015-03-16 23:40:31,775 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: automaton-1.11-8.jar 2015-03-16 23:40:31,884 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS 2015-03-16 23:40:32,359 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress 2015-03-16 23:40:32,365 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.merge.percent to 0.66 from MR setting mapreduce.reduce.shuffle.merge.percent 2015-03-16 23:40:32,366 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.fetch.buffer.percent to 0.70 from MR setting mapreduce.reduce.shuffle.input.buffer.percent 2015-03-16 23:40:32,367 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.mb to 100 from MR setting mapreduce.task.io.sort.mb 2015-03-16 23:40:32,367 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.memory.limit.percent to 0.25 from MR setting mapreduce.reduce.shuffle.memory.limit.percent 2015-03-16 23:40:32,368 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.factor to 10 from MR setting mapreduce.task.io.sort.factor 2015-03-16 23:40:32,368 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.connect.timeout to 180000 from MR setting mapreduce.reduce.shuffle.connect.timeout 2015-03-16 23:40:32,369 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.internal.sorter.class to org.apache.hadoop.util.QuickSort from MR setting map.sort.class 2015-03-16 23:40:32,370 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.merge.progress.records to 10000 from MR setting mapreduce.task.merge.progress.records 2015-03-16 23:40:32,370 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress to false from MR setting mapreduce.map.output.compress 2015-03-16 23:40:32,371 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.sort.spill.percent to 0.80 from MR setting mapreduce.map.sort.spill.percent 2015-03-16 23:40:32,372 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.ssl.enable to false from MR setting mapreduce.shuffle.ssl.enabled 2015-03-16 23:40:32,373 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead to true from MR setting mapreduce.ifile.readahead 2015-03-16 23:40:32,373 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.parallel.copies to 5 from MR setting mapreduce.reduce.shuffle.parallelcopies 2015-03-16 23:40:32,374 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead.bytes to 4194304 from MR setting mapreduce.ifile.readahead.bytes 2015-03-16 23:40:32,374 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.task.input.post-merge.buffer.percent to 0.0 from MR setting mapreduce.reduce.input.buffer.percent 2015-03-16 23:40:32,375 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.read.timeout to 180000 from MR setting mapreduce.reduce.shuffle.read.timeout 2015-03-16 23:40:32,375 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress.codec to org.apache.hadoop.io.compress.DefaultCodec from MR setting mapreduce.map.output.compress.codec 2015-03-16 23:40:32,707 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - For vertex - scope-34: parallelism=1, memory=1024, java opts=-Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA 2015-03-16 23:40:33,187 [PigTezLauncher-0] INFO org.apache.pig.tools.pigstats.tez.TezScriptState - Pig script settings are added to the job 2015-03-16 23:40:33,899 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Tez Client Version: [ component=tez-api, version=0.5.2, revision=5d728786b681e549a6f33375991dcce97cbf79f3, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTIme=20141104-1809 ] 2015-03-16 23:40:34,098 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8032 2015-03-16 23:40:34,923 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Session mode. Starting session. 2015-03-16 23:40:34,923 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Session mode. Starting session. 2015-03-16 23:40:34,924 [PigTezLauncher-0] INFO org.apache.tez.client.TezClientUtils - Using tez.lib.uris value from configuration: hdfs://localhost:9000/apps/tez-0.6.0-SNAPSHOT/tez-0.6.0.tar.gz 2015-03-16 23:40:35,129 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Tez system stage directory hdfs://localhost:9000/tmp/temp-1303134320/.tez/application_1426563447689_0001 doesn't exist and is created 2015-03-16 23:40:36,048 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1426563447689_0001 2015-03-16 23:40:36,055 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - The url to track the Tez Session: http://optimf:8088/proxy/application_1426563447689_0001/ 2015-03-16 23:40:59,709 [PigTezLauncher-0] ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Cannot submit DAG org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:603) at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:100) at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:195) at org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:159) at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:167) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-03-16 23:41:00,014 [main] INFO org.apache.pig.tools.pigstats.tez.TezPigScriptStats - Script Statistics: HadoopVersion: 2.6.0 PigVersion: 0.14.0 TezVersion: 0.5.2 UserId: dagriq FileName: dev2/email4.pig StartedAt: 2015-03-16 23:40:28 FinishedAt: 2015-03-16 23:41:00 Features: FILTER Failed! DAG PigLatin:dummy data-0_scope-0: ApplicationId: null TotalLaunchedTasks: -1 FileBytesRead: -1 FileBytesWritten: -1 HdfsBytesRead: 0 HdfsBytesWritten: 0 Input(s): Output(s): 2015-03-16 23:41:00,036 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message Details at logfile: /usr/local/pig/logs/pig-err.log 2015-03-16 23:41:00,146 [main] INFO org.apache.pig.Main - Pig script completed in 42 seconds and 897 milliseconds (42897 ms) 2015-03-16 23:41:00,147 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting down thread pool What is wrong in my installation ? Thank you for your help!
