Thanks Roshan. But issue is now resolved. I compiled flume 1.4 from source using protobuf 2.5 and this solved the below mentioned issue. My guess is that the distribution available for flume 1.4 from the release site must have been compiled using protobuf 2.4 Regards, Wesley Date: Mon, 28 Oct 2013 13:41:52 -0700 Subject: Re: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink From: [email protected] To: [email protected]
HDP already includes hadoop 2.2.0 + Flume 1.4. You'd be better of just using instead of building and testing that combo out on your own. http://hortonworks.com/products/hdp-2/?b=2#overview On Mon, Oct 28, 2013 at 1:19 PM, wesley dias <[email protected]> wrote: Hi David, Thanks for the quick reply.Does this mean that I would need to compile the flume from source using protobuf 2.5 instead of directly using the flume distribution available for download from the site? Is flume 1.4 based of protobuf 2.4? I am not sure of the same hence I am asking these questions. Regards, Wesley Date: Mon, 28 Oct 2013 16:11:16 -0400 Subject: Re: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" on writing to HDFS sink From: [email protected] To: [email protected] Looks like you have a protocol buffer version problem. Do you have a dependency on 2.5.0 version of protocol buffers in your pom? see http://code.google.com/p/protobuf/issues/detail?id=493 On Mon, Oct 28, 2013 at 4:02 PM, wesley dias <[email protected]> wrote: Hi, I am using the latest flume build 1.4 along with hadoop 2.2.0 to collect twitter stream and write to HDFS sink. However, on writing to HDFS I get the following error: 2013-10-28 09:41:23,481 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:422)] process failed java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$SetOwnerRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2531) at java.lang.Class.privateGetPublicMethods(Class.java:2651) at java.lang.Class.privateGetPublicMethods(Class.java:2661) at java.lang.Class.getMethods(Class.java:1467) I am currently new to flume and any help shall be really appreciated. Regards, Wesley CONFIDENTIALITY NOTICENOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
