Hi, Would it be possible to write directly to the Flume TCP socket using the binary AVRO format? This could be an option in our use-case.
Thanks, Dennis Von: Eric Sammer <[email protected]<mailto:[email protected]>> Antworten an: <[email protected]<mailto:[email protected]>> Datum: Wed, 19 Oct 2011 10:55:38 -0700 An: <[email protected]<mailto:[email protected]>> Betreff: Re: Using Flume agent on binary Avro file Dennis: There's no good way to tail an Avro file without writing a custom source. Your options are probably to either move the complete file (without Flume) or have the application write directly to the Avro source (using the log4j appender or something similar). On Wed, Oct 19, 2011 at 4:22 AM, Meyer, Dennis <[email protected]<mailto:[email protected]>> wrote: Hi, We've got some use case I couldn't find good information about: We use AVRO binary logging format and log the data out in binary compressed AVRO format as we want to kep the data small. Those files we'd like a Flume agent to listen and stream the data into hadoop. The current process needs the intermediate file – so no direct to Hadoop streaming is possible. Is there any way to do this, would wonder if it's not supported, but can't find infos? Thanks, Dennis -- Eric Sammer twitter: esammer data: www.cloudera.com<http://www.cloudera.com>
