On Tue, 8 Oct 2013, Radu Gheorghe wrote:
2013/10/8 David Lang <[email protected]>
The big problem with this idea is that in general messages do not stay in
the queues for very long. They are there only until they can get delivered
to their destinations.
If you were to have rsyslog stop delivering the messages to a destination
and then hang on to them all, how would rsyslog ever know if all the
destinations had completed their polling?
Ah, I think there's a misunderstanding here. I'm not suggesting to give
access to any action's queue. I'm suggesting to have a separate queue for
the REST API. For example, rsyslog is listening to /dev/log, and instead of
forwarding those messages via TCP, put them in the action queue and wait
for something to pull those messages from the queue. When once asks for a
batch of messages, they get "consumed".
It is easy enough to have rsyslog deliver messages to a queue of your
choice (may I suggest that you look at either Zero MQ or Rabbit MQ which
are already supported by rsyslog), and then you can deal with them at your
convienience.
Right, thanks for mentioning. This will work as well. Although having the
REST API directly in rsyslog will eliminate the need of another moving
piece.
well, it's another moving piece in any case, whether you call it part of rsyslog
or separate :-)
There's also the problem that giving outside programs access to the
rsyslog internals would be fragile, it wouldn't work with any other syslog
daemon (destroying the value of standardization), and it would probably not
continue to work even with rsyslog in the face of updates or even config
changes.
Ah, but this has nothing to do with syslog standards. Think of it as a
different output, like omelasticsearch. Only instead of pushing logs,
rsyslog will store them so they can get pulled.
ahh, but anything that pretends to be serious about dealing with logs can handle
syslog, if you create a new mechanism, then you will have to modify every tool
to deal with the new protocol and format. It's still lots of new modules to
write, just now it's a now module for each piece of software that will consume
the logs rather than a module for rsyslog to feed it to the other software more
efficently.
As for the logstash recommendations, I can tell you that the idea of
putting all your logs through a single chain of processing the way they do
doesn't scale that well. At really high log volumes, you need to be able to
split your processing not only across different servers, but across
different farms of servers.
The "ASCII diagram" was just a sketch of the flow, I didn't include scaling
in there. With Logstash in particular, you can have multiple "receivers"
across multiple servers, then if you use Redis as a buffer you can cluster
that, and you can have multiple Logstash "indexers" to send logs to
Elasticsearch, which can be clustered, too.
Maybe my drawing was confusing, sorry for that. I wanted to show the chain
of processing in terms of technology, not in terms of scale.
but even if there are multiple systems in each set, the idea of having all the
logs go to one set of systems, then get forwarded to the next set of systems,
etc is not really the best way to go.
Yes, you want to get all the logs funneled into one set of central systems, but
from there you want to fan out to multiple different destinations as quickly as
possible, one destination may be your elasticsearch cluster, another may be a
simple archive to disk, while another is an event correlation cluster, etc.
take a look at the architecture I outline in this paper
https://www.usenix.org/conference/lisa12/building-100k-logsec-logging-infrastructure
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
