Thank you Shekhar, Harsh. I will follow and try to implement your suggestions. I appreciate the help.
On Sat, Aug 31, 2013 at 11:12 AM, Harsh J <[email protected]> wrote: > Your cluster is using HDFS HA, and therefore requires a little more > configs than just fs.defaultFS/etc.. > > You need to use the right set of cluster client configs. If you don't > have them at /etc/hadoop/conf and /etc/hbase/conf on your cluster edge > node to pull from, try asking your cluster administrator for a > configuration set, and place their parent directories on your > application's classpath. > > The first error deals with perhaps you also including a guava > dependency in your project, which is different than the one > transitively pulled in by hadoop-client via crunch. You should be able > to use guava libs without needing an explicit dependency, and it would > be the right needed version. > > The second error deals with your MR submission failing, cause the JT > is using a staging directory over a HDFS HA, which uses a "logical" > name of "bdatadev". A logical HA name needs other configs (typically > in the hdfs-site.xml) that tell it which are the actual physical NNs > under it - configs that you're missing here. > > On Sat, Aug 31, 2013 at 1:34 AM, Narlin M <[email protected]> wrote: > > I am getting following exception while trying to submit a crunch pipeline > > job to a remote hadoop cluster: > > > > Exception in thread "main" java.lang.RuntimeException: Cannot create job > > output directory /tmp/crunch-324987940 > > at > > > org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:344) > > at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:125) > > at test.CrunchTest.setup(CrunchTest.java:98) > > at test.CrunchTest.main(CrunchTest.java:367) > > Caused by: java.io.IOException: Failed on local exception: > > com.google.protobuf.InvalidProtocolBufferException: Protocol message > > end-group tag did not match expected tag.; Host Details : local host is: > > "NARLIN/127.0.0.1"; destination host is: "<server_address>":50070; > > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:759) > > at org.apache.hadoop.ipc.Client.call(Client.java:1164) > > at > > > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) > > at com.sun.proxy.$Proxy11.mkdirs(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) > > at > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) > > at com.sun.proxy.$Proxy11.mkdirs(Unknown Source) > > at > > > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:425) > > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1943) > > at > > > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:523) > > at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1799) > > at > > > org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:342) > > ... 3 more > > Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol > > message end-group tag did not match expected tag. > > at > > > com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:73) > > at > > > com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) > > at > > > com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:213) > > at > > > com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:746) > > at > > > com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238) > > at > > > com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:282) > > at > > > com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760) > > at > > > com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288) > > at > > > com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752) > > at > > > org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcPayloadHeaderProtos.java:985) > > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:882) > > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:813) > > 0 [Thread-3] WARN org.apache.hadoop.util.ShutdownHookManager - > > ShutdownHook 'ClientFinalizer' failed, java.lang.NoSuchMethodError: > > com.google.common.collect.LinkedListMultimap.values()Ljava/util/List; > > java.lang.NoSuchMethodError: > > com.google.common.collect.LinkedListMultimap.values()Ljava/util/List; > > at org.apache.hadoop.hdfs.SocketCache.clear(SocketCache.java:135) > > at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:672) > > at > > > org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539) > > at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2308) > > at > > > org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2324) > > at > > > org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) > > > > Google search on this error yielded solutions that asked to confirm that > > /etc/hosts file contained the entry for NARLIN which it does in my case. > > > > Here's the code that I am using to set up the MRPipeline: > > > > Configuration conf = HBaseConfiguration.create(); > > > > conf.set("fs.defaultFS", "hdfs://<server_address>:50070"); > > conf.set("mapred.job.tracker", "<server_address>:50030"); > > > > System.out.println("Hadoop configuration created."); > > System.out.println("Initializing crunch pipeline ..."); > > > > conf.set("mapred.jar", "<path_to_jar_file>"); > > > > pipeline = new MRPipeline(getClass(), "crunchjobtest", conf); > > > > Has anyone faced this issue before and knows how to resolve it/point out > if > > I am missing anything? > > > > Thanks for the help. > > > > -- > Harsh J >
