I follow this blog and comment <!--<module>tez-yarn-timeline-history-with-acls</module>--> in tez-plugins/pom.xml. Build Successful.
Run successful [hdfs@test jars]$ hadoop jar tez-examples-0.7.0.jar orderedwordcount /input/inputfile.txt /tmp SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/tez0.7/jars/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 15/06/04 15:45:27 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.7.0, revision=${buildNumber}, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150604-1537 ] 15/06/04 15:45:27 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032 15/06/04 15:45:28 INFO examples.OrderedWordCount: Running OrderedWordCount 15/06/04 15:45:28 INFO client.TezClient: Submitting DAG application with id: application_1433398549746_0006 15/06/04 15:45:28 INFO client.TezClientUtils: Using tez.lib.uris value from configuration: hdfs:///apps/tez0.7/tez-0.7.0.tar.gz 15/06/04 15:45:28 INFO client.TezClient: Tez system stage directory hdfs://localhost:8020/tmp/hdfs/tez/staging/.tez/application_1433398549746_0006 doesn't exist and is created 15/06/04 15:45:28 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1433398549746_0006, dagName=OrderedWordCount 15/06/04 15:45:28 INFO impl.YarnClientImpl: Submitted application application_1433398549746_0006 15/06/04 15:45:28 INFO client.TezClient: The url to track the Tez AM: http://localhost:8088/proxy/application_1433398549746_0006/ 15/06/04 15:45:28 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032 15/06/04 15:45:29 INFO client.DAGClientImpl: Waiting for DAG to start running 15/06/04 15:45:37 INFO client.DAGClientImpl: DAG initialized: CurrentState=Running 15/06/04 15:45:37 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0 15/06/04 15:45:37 INFO client.DAGClientImpl: VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0 15/06/04 15:45:37 INFO client.DAGClientImpl: VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0 15/06/04 15:45:37 INFO client.DAGClientImpl: VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0 15/06/04 15:45:41 INFO client.DAGClientImpl: DAG: State: RUNNING Progress: 33.33% TotalTasks: 3 Succeeded: 1 Running: 1 Failed: 0 Killed: 0 15/06/04 15:45:41 INFO client.DAGClientImpl: VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0 15/06/04 15:45:41 INFO client.DAGClientImpl: VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 1 Failed: 0 Killed: 0 15/06/04 15:45:41 INFO client.DAGClientImpl: VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0 15/06/04 15:45:41 INFO client.DAGClientImpl: DAG: State: SUCCEEDED Progress: 100% TotalTasks: 3 Succeeded: 3 Running: 0 Failed: 0 Killed: 0 15/06/04 15:45:41 INFO client.DAGClientImpl: VertexStatus: VertexName: Tokenizer Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0 15/06/04 15:45:41 INFO client.DAGClientImpl: VertexStatus: VertexName: Summation Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0 15/06/04 15:45:41 INFO client.DAGClientImpl: VertexStatus: VertexName: Sorter Progress: 100% TotalTasks: 1 Succeeded: 1 Running: 0 Failed: 0 Killed: 0 15/06/04 15:45:41 INFO client.DAGClientImpl: DAG completed. FinalState=SUCCEEDED [email protected] From: [email protected] Date: 2015-06-04 15:13 To: user Subject: Re: Re: hive 1.1.0 tez0.7 hadoop 2.5.0 run query NoClassDefFoundError Thank you! [email protected] From: Jianfeng (Jeff) Zhang Date: 2015-06-04 14:41 To: user Subject: Re: hive 1.1.0 tez0.7 hadoop 2.5.0 run query NoClassDefFoundError Maybe this blog is helpful for you http://www.swiss-scalability.com/2015/05/hive-on-tez-on-cdh-love-everywhere.html Best Regard, Jeff Zhang From: "[email protected]" <[email protected]> Reply-To: user <[email protected]> Date: Thursday, June 4, 2015 at 2:05 PM To: user <[email protected]> Subject: Re: Re: hive 1.1.0 tez0.7 hadoop 2.5.0 run query NoClassDefFoundError ok I will try . My hadoop version is Hadoop 2.5.0-cdh5.2.0 Subversion http://github.com/cloudera/hadoop -r e1f20a08bde76a33b79df026d00a0c91b2298387 Compiled by jenkins on 2014-10-11T21:00Z Compiled with protoc 2.5.0 From source with checksum 309bccd135b199bdfdd6df5f3f4153d This command was run using /opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/jars/hadoop-common-2.5.0-cdh5.2.0.jar [email protected] From: Jianfeng (Jeff) Zhang Date: 2015-06-04 14:00 To: user Subject: Re: hive 1.1.0 tez0.7 hadoop 2.5.0 run query NoClassDefFoundError I believe you are using CDH. I remember there’s one thread in hive mail list also mention this issue. If I remember it correctly, you need to rebuild tez with CDH hadoop version. Best Regard, Jeff Zhang From: "[email protected]" <[email protected]> Reply-To: user <[email protected]> Date: Thursday, June 4, 2015 at 1:54 PM To: user <[email protected]> Subject: Re: hive 1.1.0 tez0.7 hadoop 2.5.0 run query NoClassDefFoundError I commet <!-- <property> <description>Log history using the Timeline Server</description> <name>tez.history.logging.service.class</name> <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value> </property> <property> <description>Publish configuration information to Timeline server.-XX:+UseParallelGC -XX:+UseG1GC </description> <name>tez.runtime.convert.user-payload.to.history-text</name> <value>true</value> </property>--> The problem not occur. Then I try run hadoop jar tez-examples-0.7.0.jar orderedwordcount /usr/tez/tez-site.xml /tmp . Return error: Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) I check log for detail ,I found: 2015-06-04 13:46:04,218 INFO [main] app.DAGAppMaster: Running DAG: OrderedWordCount 2015-06-04 13:46:04,263 ERROR [main] app.DAGAppMaster: Error starting DAGAppMaster java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native Method) at org.apache.hadoop.util.NativeCrc32.calculateChunkedSumsByteArray(NativeCrc32.java:86) at org.apache.hadoop.util.DataChecksum.calculateChunkedSums(DataChecksum.java:430) at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:202) at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:163) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1993) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1946) at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130) at org.apache.tez.dag.history.recovery.RecoveryService.handle(RecoveryService.java:286) at org.apache.tez.dag.history.HistoryEventHandler.handleCriticalEvent(HistoryEventHandler.java:102) at org.apache.tez.dag.app.DAGAppMaster.startDAG(DAGAppMaster.java:2153) at org.apache.tez.dag.app.DAGAppMaster.startDAG(DAGAppMaster.java:2103) at org.apache.tez.dag.app.DAGAppMaster.serviceStart(DAGAppMaster.java:1798) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.tez.dag.app.DAGAppMaster$6.run(DAGAppMaster.java:2224) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2220) at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2038) [email protected] From: [email protected] Date: 2015-06-04 13:19 To: user Subject: hive 1.1.0 tez0.7 hadoop 2.5.0 run query NoClassDefFoundError hive> set hive.execution.engine=tez; hive> set hive.tez.exec.print.summary=true; hive> select userid,count(*) from u_data group by userid order by userid; Query ID = hdfs_20150604131717_6863a8c8-c3a6-4865-b098-06aff509dbf8 Total jobs = 1 Launching Job 1 out of 1 java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/api/records/timeline/TimelineDomain at org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.createTimelineDomain(ATSHistoryACLPolicyManager.java:120) at org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.createSessionDomain(ATSHistoryACLPolicyManager.java:167) at org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.setupSessionACLs(ATSHistoryACLPolicyManager.java:228) at org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:525) at org.apache.tez.client.TezClient.start(TezClient.java:352) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1638) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1397) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.api.records.timeline.TimelineDomain at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 26 more FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask. org/apache/hadoop/yarn/api/records/timeline/TimelineDomain hive> [email protected]
