Are you running your Hadoop cluster in Kerberos mode ? If so, is your kerberos principal/keytab combination correct ? You can try to login into the KDC server of your hadoop cluster independently using the specified principal/keytab to make sure the combination can login to the KDC/cluster and use the above combination in your flume config.
Thanks, Rufus On Thu, May 14, 2015 at 7:04 PM, 鹰 <[email protected]> wrote: > > thanks Hari, > when i use avor client send data to the flume port I > get an kerberos error ,it looks like this : > [ERROR - > org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:561)] > Hadoop running in secure mode, but Flume config doesn't specify a principal > to use for Kerberos auth. > [ERROR - > org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:273)] > Failed to authenticate! > and then I add the hdfs.kerberosPrincipal to the configure file but it > dosen't work , what can I do now? > thanks for any help . > > > ------------------ 原始邮件 ------------------ > *发件人:* "Hari Shreedharan";<[email protected]>; > *发送时间:* 2015年5月14日(星期四) 中午11:55 > *收件人:* "user"<[email protected]>; > *主题:* Re: set flume send logs to hdfs error > > You need to use Flume’s client API to send data to the Avro Source. Use > the thrift source, and generate thrift client for Python using this IDL > file: > https://github.com/apache/flume/blob/trunk/flume-ng-sdk/src/main/thrift/flume.thrift > > You can use that to send data to the thrift source at that point. > > Thanks, > Hari Shreedharan > > > > > On May 13, 2015, at 8:06 PM, 鹰 <[email protected]> wrote: > > I send data by python scripts use socket send the code like this: > > import sys > from socket import * > > > HOST = > '192.168.1.117' > > PORT =44444 > BUFSIZ = 1024 > ADDR = (HOST, PORT) > > tcpCliSock = socket(AF_INET, SOCK_STREAM) > tcpCliSock.connect(ADDR) > i=0 > for x in range(3): > print x, "xx" > n=tcpCliSock.send("test datas from flume") > tcpCliSock.close() > > > > ------------------ 原始邮件 ------------------ > *发件人:* "Hari Shreedharan";<[email protected]>; > *发送时间:* 2015年5月14日(星期四) 上午10:53 > *收件人:* "[email protected]"<[email protected]>; > *主题:* Re: set flume send logs to hdfs error > > How are you sending data to the Avro Source? > > > Thanks, > Hari > > On Wed, May 13, 2015 at 7:38 PM, 鹰 <[email protected]> wrote: > >> hi all , >> i'm want set flume send data to hdfs my configure file is lile this : >> tier1.sources=source1 >> tier1.channels=channel1 >> tier1.sinks=sink1 >> >> tier1.sources.source1.type=avro >> tier1.sources.source1.bind=0.0.0.0 >> tier1.sources.source1.port=44444 >> tier1.sources.source1.channels=channel1 >> >> tier1.channels.channel1.type=memory >> tier1.channels.channel1.capacity=10000 >> tier1.channels.channel1.transactionCapacity=1000 >> tier1.channels.channel1.keep-alive=30 >> >> tier1.sinks.sink1.type=hdfs >> tier1.sinks.sink1.channel=channel1 >> tier1.sinks.sink1.hdfs.path=hdfs://hadoop-home.com:9000/user/hadoop/ >> tier1.sinks.sink1.hdfs.fileType=DataStream >> tier1.sinks.sink1.hdfs.writeFormat=Text >> tier1.sinks.sink1.hdfs.rollInterval=0 >> tier1.sinks.sink1.hdfs.rollSize=10240 >> tier1.sinks.sink1.hdfs.rollCount=0 >> tier1.sinks.sink1.hdfs.idleTimeout=60 >> >> when I start the flume by this configure file and send data to the port >> 44444 I get an error : >> org.apache.avro.AvroRuntimeException: Excessively large list allocation >> request detected: 154218761 items! Connection closed; >> dose anybody can help me ,thanks. >> > > >
