At LinkedIn we do this from within the application. Our reason is that we send all messsages as structured Avro, which allows us to automatically carry this schema through to Hadoop/Hive/Pig and live consumers. In other words we don't use syslog.
However, since much of the world uses syslog not our custom avro stuff it would be great to have some kind of syslog support. If anyone is interested in contributing such a thing we would be very interested to take it. Not sure if folks have ideas on the best way to support it... -Jay On Fri, May 4, 2012 at 2:29 AM, navneet sharma <navneetsharma0...@gmail.com> wrote: > I am working on a similar use-case. > I think there are 2 approaches which you can think of: > 1) Application generating syslog should have a parallel producer to send > syslog to kafka broker. > 2) try the kafka log4j appender, but again it will be in parallel with > syslogs > > Alternatively, i am trying with file APIs - infinite while loop on the > syslog file, but that is very inefficient. > > -navneet sharma > > On Tue, Apr 24, 2012 at 1:31 AM, Flinkster <flinks...@gmail.com> wrote: > >> How does LinkedIn handle syslog? Is Kafka at LinkedIN dedicated to >> application logs & metrics and syslog messages (e.g. network, OS logs) >> is a separate flow and datastore? >> >> On Mon, Apr 23, 2012 at 11:26 AM, Joel Koshy <jjkosh...@gmail.com> wrote: >> > This is an interesting use-case and similar to using the log4j appender >> for >> > application logs. What you describe sounds reasonable: i.e., have a >> > producer process on the syslog server to send syslog messages to your >> Kafka >> > brokers. >> > >> > >> >> 2) How about tailing a file to a central logging like you can do with >> >> scribe/flume agents? >> >> >> > >> > You could use tail and pipe (or a named pipe) to a console-based >> producer. >> > I don't know enough about syslog.d to tell if this would be too much of a >> > hack or not, but I think it would be more reliable than using tail+pipes: >> > you could configure your local machines to direct syslog messages to the >> > remote server and you could write a simple component that listens on the >> > syslog socket and redirects incoming messages to a Kafka producer. >> > >> > Joel >>
