Hi, we are running into a strange problem using Flume-NG 1.10 from CDH 4.0.1.
Setup:
Flume-NG opens a TCP syslog port, collects all messages and forwards them
directly into HDFS. This works fine until the point where we want to forward MS
IIS Logs in W3C format. The reason seems to be a " - " inside the log message.
I could reproduce the problem using rsyslogd forwarding all syslog messages to
flume:
logger "Hello this is a test" => Works fine :)
logger "hello - this will break" => breaks flume :(
If I remove the time variables from the HDFS path in our configuration
(attached) everything is working fine...
Exception:
2012-07-11 11:08:18,292 ERROR hdfs.HDFSEventSink: process failed
java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:375)
at java.lang.Long.valueOf(Long.java:525)
at
org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
at
org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
2012-07-11 11:08:18,294 ERROR flume.SinkRunner: Unable to deliver event.
Exception follows.
org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: null
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:375)
at java.lang.Long.valueOf(Long.java:525)
at
org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
at
org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
... 3 more
I attached our configuration in case something is broken in there.
Best regards,
Christian Schroer
--
AutoScout24 GmbH
Dingolfinger Str. 1-15
81673 München
E-Mail [email protected]
Internet www.autoscout24.de
________________________________________________________________________________________
Geschäftsführer: Alberto Sanz de Lama, André Stark
Vorsitzender des Beirats: Dr. Martin Enderle | Amtsgericht München HRB 128 701
example.conf
Description: example.conf
