>> tez.lib.uris assumes that paths are based on the default fs and therefore if >> your setup is using HDFS as default, the paths /usr/lib would be invalid
Are you sure? The below paths looks right to me except that the contents of the directories are wrong. <name>tez.lib.uris</name> <value>/usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib</value> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls /usr/lib/apache-tez-0.7.1-bin Found 2 items -rw-r--r-- 2 hduser supergroup 53092828 2016-05-20 23:15 /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz drwxr-xr-x - hduser supergroup 0 2016-05-20 23:27 /usr/lib/apache-tez-0.7.1-bin/lib -----Original Message----- From: Hitesh Shah [mailto:[email protected]] Sent: Friday, May 20, 2016 4:18 PM To: [email protected] Subject: Re: My first TEZ job fails Can you try the instructions mentioned at https://cwiki.apache.org/confluence/display/TEZ/Tez+Release+FAQ ? tez.lib.uris assumes that paths are based on the default fs and therefore if your setup is using HDFS as default, the paths /usr/lib would be invalid. — HItesh > On May 20, 2016, at 3:39 PM, Mich Talebzadeh <[email protected]> > wrote: > > Still failing with /apache-tez-0.7.1-bin I am afraid. > > OK this is my tez-site.xml > > hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml > <configuration> > <property> > <name>tez.version</name> > <value>0.7.1</value> > </property> > <property> > <name>tez.lib.uris</name> > > <value>/usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib</value> > </property> > </configuration> > > This is what I have put in hdfs directory > > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls > /usr/lib/apache-tez-0.7.1-bin Found 2 items > -rw-r--r-- 2 hduser supergroup 53092828 2016-05-20 23:15 > /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz > drwxr-xr-x - hduser supergroup 0 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib > > Also I put all /usr/lib/apache-tez-0.7.1-bin/lib/*.jar in > /usr/lib/apache-tez-0.7.1-bin/lib > > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls > /usr/lib/apache-tez-0.7.1-bin/lib Found 22 items > -rw-r--r-- 2 hduser supergroup 124846 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/RoaringBitmap-0.4.9.jar > -rw-r--r-- 2 hduser supergroup 41123 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/commons-cli-1.2.jar > -rw-r--r-- 2 hduser supergroup 58160 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/commons-codec-1.4.jar > -rw-r--r-- 2 hduser supergroup 588337 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/commons-collections-3.2.2.jar > -rw-r--r-- 2 hduser supergroup 751238 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/commons-collections4-4.1.jar > -rw-r--r-- 2 hduser supergroup 185140 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/commons-io-2.4.jar > -rw-r--r-- 2 hduser supergroup 284220 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/commons-lang-2.6.jar > -rw-r--r-- 2 hduser supergroup 1599627 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/commons-math3-3.1.1.jar > -rw-r--r-- 2 hduser supergroup 1648200 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/guava-11.0.2.jar > -rw-r--r-- 2 hduser supergroup 664918 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-common-2.6.0.jar > -rw-r--r-- 2 hduser supergroup 1509399 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-core-2.6.0.jar > -rw-r--r-- 2 hduser supergroup 130458 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/jersey-client-1.9.jar > -rw-r--r-- 2 hduser supergroup 147952 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/jersey-json-1.9.jar > -rw-r--r-- 2 hduser supergroup 81743 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/jettison-1.3.4.jar > -rw-r--r-- 2 hduser supergroup 539912 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/jetty-6.1.26.jar > -rw-r--r-- 2 hduser supergroup 177131 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/jetty-util-6.1.26.jar > -rw-r--r-- 2 hduser supergroup 33031 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/jsr305-2.0.3.jar > -rw-r--r-- 2 hduser supergroup 111908 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/metrics-core-3.1.0.jar > -rw-r--r-- 2 hduser supergroup 533455 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/protobuf-java-2.5.0.jar > -rw-r--r-- 2 hduser supergroup 105112 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/servlet-api-2.5.jar > -rw-r--r-- 2 hduser supergroup 26084 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/slf4j-api-1.7.5.jar > -rw-r--r-- 2 hduser supergroup 8869 2016-05-20 23:27 > /usr/lib/apache-tez-0.7.1-bin/lib/slf4j-log4j12-1.7.5.jar > > Still getting this error > > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hadoop jar > ./tez-examples-0.7.1.jar orderedwordcount /tmp/input/test.txt /tmp/out > > Application application_1463775706014_0008 failed 2 times due to Error > launching appattempt_1463775706014_0008_000002. Got exception: > org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError): > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J) > I at > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(Ya > rnProtos.java:2616) at > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hash > Code(YarnProtos.java:3154) at > org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifier > Proto.hashCode(YarnSecurityTokenProtos.java:410) > at > org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIden > tifier.java:126) at java.util.HashMap.hash(HashMap.java:338) > > My protoc is 2.6.1 but everything else Hadoop, spark, hive, etx all work fine! > > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> protoc --version > libprotoc 2.6.1 > > Thanks > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCC > dOABUrV8Pw > > http://talebzadehmich.wordpress.com > > > On 20 May 2016 at 23:04, Mich Talebzadeh <[email protected]> wrote: > Thanks Bikas, > > Downloaded it and installed under /usr/lib/apache-tez-0.7.1-bin > > Are these env variables correct? > > export TEZ_HOME=/usr/lib/apache-tez-0.7.1-bin > export TEZ_JARS=/usr/lib/apache-tez-0.7.1-bin > > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCC > dOABUrV8Pw > > http://talebzadehmich.wordpress.com > > > On 20 May 2016 at 22:43, Bikas Saha <[email protected]> wrote: > Yes. Almost all Hadoop ecosystem components are on 2.5.1 for protoc. > > > > Btw, unless you need something specific from 0.8 branch and are on the right > Hadoop version 2.6.0 or higher, you could use the recent 0.7.1 binary release > so you wont have to build the code yourself. May save you from these mismatch > issues for the most part. > > > > Bikas > > > > From: Mich Talebzadeh [mailto:[email protected]] > Sent: Friday, May 20, 2016 2:25 PM > To: [email protected] > Subject: Re: My first TEZ job fails > > > > the issue is that I have protobuf-2.6.1 and I believe I modified the pom.xml > under apache-tez-0.8.3-src and changed it from 2.5.0 to 2.6.1. > > > > Now I have to install protobuf-2.5.0 and It is not working. > > > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCC > dOABUrV8Pw > > http://talebzadehmich.wordpress.com > > > > On 20 May 2016 at 21:46, Mich Talebzadeh <[email protected]> wrote: > > sounds like this may ve the issue > > > > java.lang.NoSuchMethodError: > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J) > I > > > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCC > dOABUrV8Pw > > http://talebzadehmich.wordpress.com > > > > On 20 May 2016 at 19:28, Mich Talebzadeh <[email protected]> wrote: > > Hi Hitesh, > > > > This is the content of tez-site.xml] > > > > hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml > <configuration> > <property> > <name>tez.version</name> > <value>0.8.3</value> > </property> > > <property> > <name>tez.lib.uris</name> > <value>/usr/lib/tez-0.8.3,/usr/lib/tez-0.8.3/lib/</value> > </property> > </configuration> > > > > > > > > This is what is in hdfs > > > > drwxr-xr-x - hduser supergroup 0 2016-05-20 19:25 > /usr/lib/tez-0.8.3/lib > -rw-r--r-- 2 hduser supergroup 42821282 2016-05-20 09:03 > /usr/lib/tez-0.8.3/tez-0.8.3.tar.gz > > > > > > Tez is installed in > > > > cd $TEZ_HOME > hduser@rhes564: /usr/lib/tez-0.8.3> > > > > tez-site.xml is soft linked in $TEZ_HOME as follows > > > > hduser@rhes564: /usr/lib/tez-0.8.3> ls -l tez-site.xml lrwxrwxrwx 1 > hduser hadoop 49 May 20 19:22 tez-site.xml -> > /home/hduser/hadoop-2.6.0/etc/hadoop/tez-site.xml > > > > This is error > > > > Application application_1463758195355_0004 failed 2 times due to Error > launching appattempt_1463758195355_0004_000002. Got exception: > org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError): > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J) > I > > at > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(Ya > rnProtos.java:2616) > > at > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hash > Code(YarnProtos.java:3154) > > at > org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifier > Proto.hashCode(YarnSecurityTokenProtos.java:410) > > at > org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIden > tifier.java:126) > > at java.util.HashMap.hash(HashMap.java:338) > > at java.util.HashMap.put(HashMap.java:611) > > at java.util.HashSet.add(HashSet.java:219) > > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409) > > at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1369) > > at javax.security.auth.Subject.getPublicCredentials(Subject.java:720) > > at > org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(Us > erGroupInformation.java:1400) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerMa > nagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerMa > nagerImpl.startContainers(ContainerManagerImpl.java:699) > > at > org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocol > PBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl > .java:60) > > at > org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerMana > gementProtocolService$2.callBlockingMethod(ContainerManagementProtocol > .java:95) > > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call > (ProtobufRpcEngine.java:619) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat > ion.java:1628) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) > > at org.apache.hadoop.ipc.Client.call(Client.java:1468) > > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngi > ne.java:232) > > at com.sun.proxy.$Proxy80.startContainers(Unknown Source) > > at > org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolP > BClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.ja > va:96) > > at > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.la > unch(AMLauncher.java:119) > > at > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.ru > n(AMLauncher.java:254) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > ava:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > java:617) > > at java.lang.Thread.run(Thread.java:745) > > . Failing the application. > > Thanks > > > > > > > > > > > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCC > dOABUrV8Pw > > http://talebzadehmich.wordpress.com > > > > On 20 May 2016 at 19:06, Hitesh Shah <[email protected]> wrote: > > Logs from `bin/yarn logs -applicationId application_1463758195355_0002` would > be more useful to debug your setup issue. The RM logs usually do not shed > much light on why an application failed. > Can you confirm that you configured tez.lib.uris correctly to point to the > tez tarball on HDFS (tez tar should be the one obtained from > tez-dist/target/tez-0.8.3.tar.gz) ? > > — Hitesh > > > > On May 20, 2016, at 10:24 AM, Mich Talebzadeh <[email protected]> > > wrote: > > > > Hi, > > > > I have just compiled and installed TEZ, trying to do a test with > > > > hadoop jar ./tez-examples-0.8.3.jar orderedwordcount > > /tmp/input/test.txt /tmp/out > > > > The job fails as follows. This is from yarn log > > > > 2016-05-20 18:19:26,945 INFO > > SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for > > appattempt_1463758195355_0002_000001 (auth:SIMPLE) > > 2016-05-20 18:19:26,950 WARN org.apache.hadoop.ipc.Server: IPC > > Server handler 0 on 59093, call > > org.apache.hadoop.yarn.api.ContainerManagementProtocolPB.startContai > > ners from 50.140.197.217:46784 Call#2 Retry#0 > > java.lang.NoSuchMethodError: > > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I > > at > > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616) > > at > > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154) > > at > > org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410) > > at > > org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126) > > at java.util.HashMap.hash(HashMap.java:338) > > at java.util.HashMap.put(HashMap.java:611) > > at java.util.HashSet.add(HashSet.java:219) > > at > > javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409) > > at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1369) > > at > > javax.security.auth.Subject.getPublicCredentials(Subject.java:720) > > at > > org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400) > > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618) > > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699) > > at > > org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60) > > at > > org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95) > > at > > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > > at > > org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) > > 2016-05-20 18:19:27,929 WARN > > org.apache.hadoop.yarn.server.nodemanager.containermanager.Container > > ManagerImpl: Event EventType: KILL_CONTAINER sent to absent > > container container_1463758195355_0002_01_000001 > > 2016-05-20 18:19:27,944 INFO > > SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for > > appattempt_1463758195355_0002_000002 (auth:SIMPLE) > > 2016-05-20 18:19:27,949 WARN org.apache.hadoop.ipc.Server: IPC > > Server handler 0 on 59093, call > > org.apache.hadoop.yarn.api.ContainerManagementProtocolPB.startContai > > ners from 50.140.197.217:46785 Call#3 Retry#0 > > java.lang.NoSuchMethodError: > > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I > > at > > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616) > > at > > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154) > > at > > org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410) > > at > > org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126) > > at java.util.HashMap.hash(HashMap.java:338) > > at java.util.HashMap.put(HashMap.java:611) > > at java.util.HashSet.add(HashSet.java:219) > > at > > javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409) > > at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1369) > > at > > javax.security.auth.Subject.getPublicCredentials(Subject.java:720) > > at > > org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400) > > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618) > > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699) > > at > > org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60) > > at > > org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95) > > at > > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > > at > > org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) > > 2016-05-20 18:19:28,931 WARN > > org.apache.hadoop.yarn.server.nodemanager.containermanager.Container > > ManagerImpl: Event EventType: KILL_CONTAINER sent to absent > > container container_1463758195355_0002_02_000001 > > 2016-05-20 18:19:28,932 WARN > > org.apache.hadoop.yarn.server.nodemanager.containermanager.Container > > ManagerImpl: Event EventType: FINISH_APPLICATION sent to absent > > application application_1463758195355_0002 > > > > Any ideas will be appreciated! > > > > Thanks > > > > Dr Mich Talebzadeh > > > > LinkedIn > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcP > > CCdOABUrV8Pw > > > > http://talebzadehmich.wordpress.com > > > > > > > > > > >
