I tried moving from HDFS to HttpFileSystem. I’m getting the HttpFileSystem not found exception. I have done the steps in the tutorial that Chris pasted below (I had done that before, but I’m not sure what is the problem). Seems like since I have the compiled file in one machine (resource manager) and I submit it and try to download from the node managers, they don’t have samza-yarn.jar (don’t know how to include it, since the run will be done in the resource manager).
Can you give me a tip on how to solve this? Thanks in advance. ps. the folder and tar.gz of the job are located in one machine alone, is that the right way to do it or do I need to replicate hello-samza in all machines to run it? On 11 Aug 2014, at 23:12, Telles Nobrega <[email protected]> wrote: > What is your suggestion here, should I keep going on this quest to fix hdfs > or should I try to run using HttpFileSystem? > On 11 Aug 2014, at 23:01, Telles Nobrega <[email protected]> wrote: > >> The port is right?? 50700. I have no idea what is happening now. >> >> On 11 Aug 2014, at 22:33, Telles Nobrega <[email protected]> wrote: >> >>> Right now the error is the following: >>> Exception in thread "main" java.io.IOException: Failed on local exception: >>> com.google.protobuf.InvalidProtocolBufferException: Protocol message >>> end-group tag did not match expected tag.; Host Details : local host is: >>> "telles-samza-master/10.1.0.79"; destination host is: >>> "telles-samza-master":50070; >>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) >>> at org.apache.hadoop.ipc.Client.call(Client.java:1410) >>> at org.apache.hadoop.ipc.Client.call(Client.java:1359) >>> at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) >>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) >>> at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) >>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source) >>> at >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:671) >>> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1746) >>> at >>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112) >>> at >>> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108) >>> at >>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) >>> at >>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108) >>> at >>> org.apache.samza.job.yarn.ClientHelper.submitApplication(ClientHelper.scala:111) >>> at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55) >>> at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48) >>> at org.apache.samza.job.JobRunner.run(JobRunner.scala:62) >>> at org.apache.samza.job.JobRunner$.main(JobRunner.scala:37) >>> at org.apache.samza.job.JobRunner.main(JobRunner.scala) >>> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol >>> message end-group tag did not match expected tag. >>> at >>> com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94) >>> at >>> com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) >>> at >>> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202) >>> at >>> com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241) >>> at >>> com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253) >>> at >>> com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259) >>> at >>> com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49) >>> at >>> org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364) >>> at >>> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1051) >>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:945) >>> >>> I feel that I’m close to making it run. Thanks for the help in advance. >>> On 11 Aug 2014, at 22:06, Telles Nobrega <[email protected]> wrote: >>> >>>> Hi, I downloaded hadoop-common-2.3.0.jar and it worked better. Now I’m >>>> having a configuration problem with my host, but it looks like the hdfs is >>>> not a problem anymore. >>>> >>>> >>>> >>>> >>>> On 11 Aug 2014, at 22:04, Telles Nobrega <[email protected]> wrote: >>>> >>>>> So, I added hadoop-hdfs-2.3.0.jar as a maven dependency. Recompiled the >>>>> project, extracted to deploy/samza and there problem still happens. I >>>>> downloaded hadoop-client-2.3.0.jar and the problems still happens, >>>>> hadoop-common is 2.2.0 does this is a problem? I will try with 2.3.0 >>>>> >>>>> Actually a lot of hadoop jars are 2.2.0 >>>>> >>>>> On 11 Aug 2014, at 21:33, Yan Fang <[email protected]> wrote: >>>>> >>>>>> <include>org.apache.hadoop:hadoop-hdfs</include> >>>>> >>>> >>> >> >
