Yes. Almost all Hadoop ecosystem components are on 2.5.1 for protoc.
Btw, unless you need something specific from 0.8 branch and are on the right Hadoop version 2.6.0 or higher, you could use the recent 0.7.1 binary release so you wont have to build the code yourself. May save you from these mismatch issues for the most part. Bikas From: Mich Talebzadeh [mailto:[email protected]] Sent: Friday, May 20, 2016 2:25 PM To: [email protected] Subject: Re: My first TEZ job fails the issue is that I have protobuf-2.6.1 and I believe I modified the pom.xml under apache-tez-0.8.3-src and changed it from 2.5.0 to 2.6.1. Now I have to install protobuf-2.5.0 and It is not working. Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> On 20 May 2016 at 21:46, Mich Talebzadeh <[email protected] <mailto:[email protected]> > wrote: sounds like this may ve the issue java.lang.NoSuchMethodError: org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> On 20 May 2016 at 19:28, Mich Talebzadeh <[email protected] <mailto:[email protected]> > wrote: Hi Hitesh, This is the content of tez-site.xml] hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml <configuration> <property> <name>tez.version</name> <value>0.8.3</value> </property> <property> <name>tez.lib.uris</name> <value>/usr/lib/tez-0.8.3,/usr/lib/tez-0.8.3/lib/</value> </property> </configuration> This is what is in hdfs drwxr-xr-x - hduser supergroup 0 2016-05-20 19:25 /usr/lib/tez-0.8.3/lib -rw-r--r-- 2 hduser supergroup 42821282 2016-05-20 09:03 /usr/lib/tez-0.8.3/tez-0.8.3.tar.gz Tez is installed in cd $TEZ_HOME hduser@rhes564: /usr/lib/tez-0.8.3> tez-site.xml is soft linked in $TEZ_HOME as follows hduser@rhes564: /usr/lib/tez-0.8.3> ls -l tez-site.xml lrwxrwxrwx 1 hduser hadoop 49 May 20 19:22 tez-site.xml -> /home/hduser/hadoop-2.6.0/etc/hadoop/tez-site.xml This is error Application application_1463758195355_0004 failed 2 times due to Error launching appattempt_1463758195355_0004_000002. Got exception: org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError): org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I at org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616) at org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154) at org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410) at org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126) at java.util.HashMap.hash(HashMap.java:338) at java.util.HashMap.put(HashMap.java:611) at java.util.HashSet.add(HashSet.java:219) at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409) at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1369) at javax.security.auth.Subject.getPublicCredentials(Subject.java:720) at org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699) at org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) at org.apache.hadoop.ipc.Client.call(Client.java:1468) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy80.startContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:119) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:254) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) . Failing the application. Thanks Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> On 20 May 2016 at 19:06, Hitesh Shah <[email protected] <mailto:[email protected]> > wrote: Logs from `bin/yarn logs -applicationId application_1463758195355_0002` would be more useful to debug your setup issue. The RM logs usually do not shed much light on why an application failed. Can you confirm that you configured tez.lib.uris correctly to point to the tez tarball on HDFS (tez tar should be the one obtained from tez-dist/target/tez-0.8.3.tar.gz) ? — Hitesh > On May 20, 2016, at 10:24 AM, Mich Talebzadeh <[email protected] > <mailto:[email protected]> > wrote: > > Hi, > > I have just compiled and installed TEZ, trying to do a test with > > hadoop jar ./tez-examples-0.8.3.jar orderedwordcount /tmp/input/test.txt > /tmp/out > > The job fails as follows. This is from yarn log > > 2016-05-20 18:19:26,945 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for appattempt_1463758195355_0002_000001 (auth:SIMPLE) > 2016-05-20 18:19:26,950 WARN org.apache.hadoop.ipc.Server: IPC Server handler > 0 on 59093, call > org.apache.hadoop.yarn.api.ContainerManagementProtocolPB.startContainers from > 50.140.197.217:46784 <http://50.140.197.217:46784> Call#2 Retry#0 > java.lang.NoSuchMethodError: > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I > at > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616) > at > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154) > at > org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410) > at > org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126) > at java.util.HashMap.hash(HashMap.java:338) > at java.util.HashMap.put(HashMap.java:611) > at java.util.HashSet.add(HashSet.java:219) > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409) > at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1369) > at javax.security.auth.Subject.getPublicCredentials(Subject.java:720) > at > org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699) > at > org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) > 2016-05-20 18:19:27,929 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Event EventType: KILL_CONTAINER sent to absent container > container_1463758195355_0002_01_000001 > 2016-05-20 18:19:27,944 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for appattempt_1463758195355_0002_000002 (auth:SIMPLE) > 2016-05-20 18:19:27,949 WARN org.apache.hadoop.ipc.Server: IPC Server handler > 0 on 59093, call > org.apache.hadoop.yarn.api.ContainerManagementProtocolPB.startContainers from > 50.140.197.217:46785 <http://50.140.197.217:46785> Call#3 Retry#0 > java.lang.NoSuchMethodError: > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I > at > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616) > at > org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154) > at > org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410) > at > org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126) > at java.util.HashMap.hash(HashMap.java:338) > at java.util.HashMap.put(HashMap.java:611) > at java.util.HashSet.add(HashSet.java:219) > at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409) > at javax.security.auth.Subject$ClassSet.<init>(Subject.java:1369) > at javax.security.auth.Subject.getPublicCredentials(Subject.java:720) > at > org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699) > at > org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60) > at > org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) > 2016-05-20 18:19:28,931 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Event EventType: KILL_CONTAINER sent to absent container > container_1463758195355_0002_02_000001 > 2016-05-20 18:19:28,932 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Event EventType: FINISH_APPLICATION sent to absent application > application_1463758195355_0002 > > Any ideas will be appreciated! > > Thanks > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > http://talebzadehmich.wordpress.com >
