Chris,
syslog is a RFC defined protocol, we support only RFC-5424 and RFC-3164
formats. Since you've to use valid syslog events it works:
echo "<13>Jun 20 12:12:12 host foo[345]: - a syslog message with -" > /tmp/foo
nc -v YOUR_IP 5140 < /tmp/foo
12/07/11 14:51:52 INFO sink.LoggerSink: Event: { headers:{Severity=5, host=a,
Facility=8} body: 73 79 73 6C 6F 67 20 6D 65 73 73 61 67 65 20 77 syslog
message w }
- Alex
p.s. ich hab Deine Mail an mich nicht gesehen, aber es ist besser an die liste
zu schreiben ;)
On Jul 11, 2012, at 12:15 PM, Juhani Connolly wrote:
> The time variables depend on the existence of a header with the key
> "timestamp". If it isn't there, it tries to parse a non-existent header to
> calculate the time, and this happens. I don't believe it has anything to do
> with the contents of your log message.
>
> For the easiest way to add the header, I would recommend trying 1.2.0 as soon
> as it is released(or you can try grabbing the current release candidate or
> even the 1.3.0 trunk which I'm running right now without any serious issues),
> and using the TimestampInterceptor there. As this is a frequent query I've
> made a jira to document this dependency properly
> https://issues.apache.org/jira/browse/FLUME-1364
>
> On 07/11/2012 06:41 PM, Christian Schroer wrote:
>> Hi,
>>
>> we are running into a strange problem using Flume-NG 1.10 from CDH 4.0.1.
>>
>> Setup:
>> Flume-NG opens a TCP syslog port, collects all messages and forwards them
>> directly into HDFS. This works fine until the point where we want to forward
>> MS IIS Logs in W3C format. The reason seems to be a " - " inside the log
>> message. I could reproduce the problem using rsyslogd forwarding all syslog
>> messages to flume:
>>
>> logger "Hello this is a test" => Works fine :)
>>
>> logger "hello - this will break" => breaks flume :(
>>
>> If I remove the time variables from the HDFS path in our configuration
>> (attached) everything is working fine...
>>
>> Exception:
>>
>> 2012-07-11 11:08:18,292 ERROR hdfs.HDFSEventSink: process failed
>> java.lang.NumberFormatException: null
>> at java.lang.Long.parseLong(Long.java:375)
>> at java.lang.Long.valueOf(Long.java:525)
>> at
>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>> at
>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>> at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Thread.java:662)
>> 2012-07-11 11:08:18,294 ERROR flume.SinkRunner: Unable to deliver event.
>> Exception follows.
>> org.apache.flume.EventDeliveryException: java.lang.NumberFormatException:
>> null
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>> at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.NumberFormatException: null
>> at java.lang.Long.parseLong(Long.java:375)
>> at java.lang.Long.valueOf(Long.java:525)
>> at
>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>> at
>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>> ... 3 more
>>
>> I attached our configuration in case something is broken in there.
>>
>> Best regards,
>>
>> Christian Schroer
>>
>
>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
