The big problem with this idea is that in general messages do not stay in the
queues for very long. They are there only until they can get delivered to their
destinations.
If you were to have rsyslog stop delivering the messages to a destination and
then hang on to them all, how would rsyslog ever know if all the destinations
had completed their polling?
It is easy enough to have rsyslog deliver messages to a queue of your choice
(may I suggest that you look at either Zero MQ or Rabbit MQ which are already
supported by rsyslog), and then you can deal with them at your convienience.
There's also the problem that giving outside programs access to the rsyslog
internals would be fragile, it wouldn't work with any other syslog daemon
(destroying the value of standardization), and it would probably not continue to
work even with rsyslog in the face of updates or even config changes.
As for the logstash recommendations, I can tell you that the idea of putting all
your logs through a single chain of processing the way they do doesn't scale
that well. At really high log volumes, you need to be able to split your
processing not only across different servers, but across different farms of
servers.
David Lang
On Tue, 8 Oct 2013, Radu Gheorghe wrote:
Hello,
I know it's a long tail of "interesting features" that's waiting to be
implemented, and adding yet another one might be... simply too much. But
this one popped in my head as being reaaally nice for more than an year -
even though I don't particularly need it - and I've never shared it. Which
also feels wrong. So here I am, sharing it :)
The idea is simple: expose a REST API that allows external applications to
pull data from queues. Something like this:
curl 'rsyslog-host:5514/queue1'
#and the answer would be something like
{
"took": 75,
"items": [
{
"msg": "hello world!",
"hostname": "another-host",
.......other properties.....
},
{
[another message would go here. X messages in total, for
queue.dequebatchsize=X]
}
]
}
Whatever gets into those JSON items would be decided by the templates, and
the module implementing this REST API would be an output module, holding
those items in its action queue.
Maybe the [high-level] implementation details above aren't so great. Please
comment if you think there are better ways of doing it. Still, I think the
idea is valueable: provide a [simple] way for transmitting messages via a
pull mechanism rather than push.
A pull mechanism has its advantages (and disadvantages, of course) over
push:
- whatever pulls those messages won't ever be overwhelmed, as it can only
pull data as fast as it can process it. I had Elasticsearch
rivers<http://www.elasticsearch.org/guide/en/elasticsearch/rivers/current/river.html>in
mind, but really, any pluggable system should be able to pull data
from
rsyslog and process it
- some use-cases require firewalls to be set up in such a way that pulling
is more convenient
- when you need to centralize logs, a setup like this seems more and more
popular (see how Logstash recommends
Redis<http://logstash.net/docs/1.2.1/tutorials/getting-started-centralized>as
buffer):
*[log producer]-->[buffer]-->[data store]*
And having rsyslog act like a dedicated buffer will allow anyone to write
plugins and services to pull those messages and process them, without being
limited to syslog protocols or to writing rsyslog modules. This should open
rsyslog (and the syslog world in general) to get better integration with
lots of data processing and monitoring tools. For example, one can stick
the impstats logs in such a queue and write a Nagios plugin to monitor
rsyslog metrics via this REST API.
It feels like most of the needed parts are already there: templates,
queues, debatchsize, discarding when the queue is full (even by severity).
Maybe only two missing parts:
- the output module for this REST API
- some way to get queues named (aliasing for queues)
Any comments, feedback, questions? Should I just stick an enhancement in
Bugzilla, for whenever there's time and/or a sponsor?
Best regards,
Radu
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
