Hi, Thanks all of you for your help! Sadly I had to choose an another way how to transfer logs even I like flume-ng a lot. We don't have enough resources to support custom "plugins" in java written program.
Filip On Wed, Oct 17, 2012 at 1:24 AM, Bhaskar V. Karambelkar <[email protected]> wrote: > Roshan, > The problem is not at the event end, it's at the serializer end. The > BodyTextSerializer which is the default, does not serialize the > headers, so no matter what headers you have or don't have, it won't > affect the output using the default serializer. > So you've 2 options, use the avro_client serializer and get an avro > data file, or write your own serializer that writes plain text, but > does serialize the headers along with the body, in may be JSON/XML > encoding etc. > > On Tue, Oct 16, 2012 at 6:48 PM, Roshan Naik <[email protected]> wrote: >> Hari, >> wouldn't a custom interceptor be more logical ? >> -roshan >> >> >> On Tue, Oct 16, 2012 at 3:36 PM, Hari Shreedharan >> <[email protected]> wrote: >>> >>> See the code for the serializers here: >>> >>> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/serialization;h=fcc07339b3cf0f5b8d1a75e978ffc1edbab28bfe;hb=HEAD >>> You can use one of these as an example to write your own. >>> >>> The configuration documentation for HDFS Sink is here: >>> http://flume.apache.org/FlumeUserGuide.html#hdfs-sink >>> You can specify a custom serializer by specifying its builder class in the >>> config param "serializer". >>> >>> >>> Thanks, >>> Hari >>> >>> -- >>> Hari Shreedharan >>> >>> On Tuesday, October 16, 2012 at 3:25 PM, Filip Slunecko wrote: >>> >>> @Roshan: thanks for suggestion, I will look into it. >>> >>> @Hari: I tried to google it up and there is not much about it. I will >>> look into it tomorrow and let you know (It' s too late for serious >>> work in my time realm :) ) >>> >>> Thanks both of you for your quick response and help! >>> >>> Filip >>> >>> On Wed, Oct 17, 2012 at 12:00 AM, Hari Shreedharan >>> <[email protected]> wrote: >>> >>> Hi Filip, >>> >>> The reason for this is that the Text serializer will only serialized the >>> body of the event, and the syslog sources write the body of the syslog >>> event >>> into the body of the flume event. The hostname/timestamp/severity etc are >>> added into the Flume Event headers. You could simply write a serializer >>> which writes out this information in the same format as you expect and you >>> will be able to see the headers in the files. You could use the Avro >>> serializer to serialize it into avro too, which will make sure the headers >>> are also written out. >>> >>> Hope this helps. >>> >>> >>> Hari >>> >>> -- >>> Hari Shreedharan >>> >>> On Tuesday, October 16, 2012 at 2:27 PM, Filip Slunecko wrote: >>> >>> Hi, >>> >>> I am trying to use syslog source and sink it to hdfs or fileroller. >>> Everything is working, but "saved" logs are without timestamp and >>> hostname information. >>> Is it possible to force flume-ng to dump those information from syslog >>> header togather with body lines? >>> >>> I am using flume-ng-agent-1.2.0+24.4-1.noarch from Cloudera repository. >>> >>> Thanks, >>> >>> Filip >>> >>> >>
