Looks like a problem in the security setup, you are probably trying to write to a secure Hadoop cluster w/o providing the kerberos credentials to the HDFS sink. HDFS and Flume should work fine in HDP 2.1. -roshan
On 2/6/15 10:50 AM, "David Novogrodsky" <[email protected]> wrote: >All, > >Thanks your your help in the past. I have a cluster managed by Ambari >1.7 using HDP 2.1. I was told that the versions of Flume and Hadopop >in HDP 2.1 have a problem. Specifically the HDFS sink does not write >to the cluster properly. I want to test that to be sure. The error I >should be getting is this: > >java.lang.VerifyError: class >org.apache.hadoop.security.proto.SecurityProtos$GetDelegationTokenRequestP >roto >overrides final method >getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > >I am starting the agent using this command line: >[root@namenode flume]# flume-ng agent --conf conf --conf-file >/etc/flume/conf/a2/flume.conf -n a2 > >But instead I am getting this error, which seems unrelated to HDFS: >15/02/06 12:31:44 ERROR source.SequenceGeneratorSource: r1 source >could not write to channel. >org.apache.flume.ChannelException: Unable to put event on required >channel: org.apache.flume.channel.MemoryChannel{name: c1} > at >org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.ja >va:275) > >What follows is more details: > >Here is my configuration: >a2.sources = r1 >a2.sinks = k1 >a2.channels = c1 > >a2.sources.r1.type = seq > >a2.sinks.sink_to_hdfs.type = hdfs >a2.sinks.sink_to_hdfs.hdfs.fileType = DataStream >a2.sinks.sink_to_hdfs.hdfs.path = /flume/events >a2.sinks.sink_to_hdfs.hdfs.filePrefix = eventlog >a2.sinks.sink_to_hdfs.hdfs.fileSuffix = .log >a2.sinks.sink_to_hdfs.hdfs.batchSize = 100 > >a2.channels.c1.type = memory >a2.channels.c1.capacity = 10000 >a2.channels.c1.transactionCapacity = 10000 >a2.channels.c1.byteCapacityBufferPercentage = 20 >a2.channels.c1.byteCapacity = 800000 > >a2.sources.r1.channels = c1 >a2.sinks.k1.channel = c1 >a2.sinks.sink_to_hdfs.type = hdfs >a2.sinks.sink_to_hdfs.hdfs.fileType = DataStream >a2.sinks.sink_to_hdfs.hdfs.path = /flume/events >a2.sinks.sink_to_hdfs.hdfs.filePrefix = eventlog >a2.sinks.sink_to_hdfs.hdfs.fileSuffix = .log >a2.sinks.sink_to_hdfs.hdfs.batchSize = 1000 > >David Novogrodsky >[email protected] >http://www.linkedin.com/in/davidnovogrodsky
