Oh, and I should add that this also relates to my other thread about binary layouts. I'd find that using binary layouts would be most useful for network appenders, though it can also be useful for when log aggregation daemons are run on every server to collect the output of log files, too.
On 28 April 2017 at 11:54, Matt Sicker <boa...@gmail.com> wrote: > Log4j has all the necessary plugins to support numerous scenarios, and > I've found three ways to support Graylog, for example: > > 1. GelfLayout + SocketAppender: send log messages straight to a Graylog > server. Simplest setup (no additional dependencies required), but not the > most reliable in theory. > 2. GelfLayout + KafkaAppender: send log messages to Kafka first, then have > a consumer on the other side ingest those into Graylog. This style is more > flexible since the layout doesn't necessarily need to be a GelfLayout, but > this is the most efficient way to handle that scenario. The cons, however, > is that a message in memory that hasn't been sent to Kafka yet can be lost > in a crash scenario, similar to the SocketAppender limitation. > 3. GelfLayout + FlumeAppender: send messages formatted for Graylog, but > persist locally first before passing off to another flume agent. I'm not > too familiar with this setup, but based on what Ralph has explained before, > this is probably the most reliable way to ensure logs are aggregated. > > Similar patterns can be followed for other services like Logstash, Splunk, > etc. > > Anyways, what I'm looking for here are suggestions on architecture here > and perhaps the desire to get a page written in the manual describing these > types of distributed system logging scenarios. If we can explain how to > follow these architectures, we can also find any inefficiencies in them to > improve on. > > -- > Matt Sicker <boa...@gmail.com> > -- Matt Sicker <boa...@gmail.com>
