Hi, I’ve been using flume-ng to ingest data into Kafka for approx 18 months. I’ve deployed agents into a multi-vendor environment and have been struggling with processing a number of syslog formats (rfc and non-rfc compliant) in a consistent way.
Looking at the SyslogUtils code influme-ng-core there are regex parsers for rfc3164 and rfc5424 message formats. Currently whenever I have to ingest events transported via syslog which are non-rfc compliant I am having to use a regex interceptor to provide the event header extraction. I can’t prove this is less efficient but reason would suggest it; especially if the source is always going to try to extract the rfc-compliant headers regardless of whether an interceptor is confgured. My question is it worth looking at the extension of supported input formats; could the source be made configurable with a library of input formats (i.e. grok dictionary as with the MorphlineSolr sink), or is this a case of moving the event formatting upstream away from flume? Thanks, Chris
signature.asc
Description: Message signed with OpenPGP using GPGMail
