thanks Hari,
when i use avor client send data to the flume port I get an
kerberos error ,it looks like this :
[ERROR -
org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:561)]
Hadoop running in secure mode, but Flume config doesn't specify a principal to
use for Kerberos auth.
[ERROR -
org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:273)]
Failed to authenticate!
and then I add the hdfs.kerberosPrincipal to the configure file but it dosen't
work , what can I do now?
thanks for any help .
------------------ ???????? ------------------
??????: "Hari Shreedharan";<[email protected]>;
????????: 2015??5??14??(??????) ????11:55
??????: "user"<[email protected]>;
????: Re: set flume send logs to hdfs error
You need to use Flume??s client API to send data to the Avro Source. Use the
thrift source, and generate thrift client for Python using this IDL file:
https://github.com/apache/flume/blob/trunk/flume-ng-sdk/src/main/thrift/flume.thrift
You can use that to send data to the thrift source at that point.
Thanks,
Hari Shreedharan
On May 13, 2015, at 8:06 PM, ?? <[email protected]> wrote:
I send data by python scripts use socket send the code like this:
import sys
from socket import *
HOST = '192.168.1.117'
PORT =44444
BUFSIZ = 1024
ADDR = (HOST, PORT)
tcpCliSock = socket(AF_INET, SOCK_STREAM)
tcpCliSock.connect(ADDR)
i=0
for x in range(3):
print x, "xx"
n=tcpCliSock.send("test datas from flume")
tcpCliSock.close()
------------------ ???????? ------------------
??????: "Hari Shreedharan";<[email protected]>;
????????: 2015??5??14??(??????) ????10:53
??????: "[email protected]"<[email protected]>;
????: Re: set flume send logs to hdfs error
How are you sending data to the Avro Source?
Thanks,
Hari
On Wed, May 13, 2015 at 7:38 PM, ?? <[email protected]> wrote:
hi all ,
i'm want set flume send data to hdfs my configure file is lile this :
tier1.sources=source1
tier1.channels=channel1
tier1.sinks=sink1
tier1.sources.source1.type=avro
tier1.sources.source1.bind=0.0.0.0
tier1.sources.source1.port=44444
tier1.sources.source1.channels=channel1
tier1.channels.channel1.type=memory
tier1.channels.channel1.capacity=10000
tier1.channels.channel1.transactionCapacity=1000
tier1.channels.channel1.keep-alive=30
tier1.sinks.sink1.type=hdfs
tier1.sinks.sink1.channel=channel1
tier1.sinks.sink1.hdfs.path=hdfs://hadoop-home.com:9000/user/hadoop/
tier1.sinks.sink1.hdfs.fileType=DataStream
tier1.sinks.sink1.hdfs.writeFormat=Text
tier1.sinks.sink1.hdfs.rollInterval=0
tier1.sinks.sink1.hdfs.rollSize=10240
tier1.sinks.sink1.hdfs.rollCount=0
tier1.sinks.sink1.hdfs.idleTimeout=60
when I start the flume by this configure file and send data to the port 44444 I
get an error :
org.apache.avro.AvroRuntimeException: Excessively large list allocation request
detected: 154218761 items! Connection closed;
dose anybody can help me ,thanks.
