Hello,
I know it's a long tail of "interesting features" that's waiting to be
implemented, and adding yet another one might be... simply too much. But
this one popped in my head as being reaaally nice for more than an year -
even though I don't particularly need it - and I've never shared it. Which
also feels wrong. So here I am, sharing it :)
The idea is simple: expose a REST API that allows external applications to
pull data from queues. Something like this:
curl 'rsyslog-host:5514/queue1'
#and the answer would be something like
{
"took": 75,
"items": [
{
"msg": "hello world!",
"hostname": "another-host",
.......other properties.....
},
{
[another message would go here. X messages in total, for
queue.dequebatchsize=X]
}
]
}
Whatever gets into those JSON items would be decided by the templates, and
the module implementing this REST API would be an output module, holding
those items in its action queue.
Maybe the [high-level] implementation details above aren't so great. Please
comment if you think there are better ways of doing it. Still, I think the
idea is valueable: provide a [simple] way for transmitting messages via a
pull mechanism rather than push.
A pull mechanism has its advantages (and disadvantages, of course) over
push:
- whatever pulls those messages won't ever be overwhelmed, as it can only
pull data as fast as it can process it. I had Elasticsearch
rivers<http://www.elasticsearch.org/guide/en/elasticsearch/rivers/current/river.html>in
mind, but really, any pluggable system should be able to pull data
from
rsyslog and process it
- some use-cases require firewalls to be set up in such a way that pulling
is more convenient
- when you need to centralize logs, a setup like this seems more and more
popular (see how Logstash recommends
Redis<http://logstash.net/docs/1.2.1/tutorials/getting-started-centralized>as
buffer):
*[log producer]-->[buffer]-->[data store]*
And having rsyslog act like a dedicated buffer will allow anyone to write
plugins and services to pull those messages and process them, without being
limited to syslog protocols or to writing rsyslog modules. This should open
rsyslog (and the syslog world in general) to get better integration with
lots of data processing and monitoring tools. For example, one can stick
the impstats logs in such a queue and write a Nagios plugin to monitor
rsyslog metrics via this REST API.
It feels like most of the needed parts are already there: templates,
queues, debatchsize, discarding when the queue is full (even by severity).
Maybe only two missing parts:
- the output module for this REST API
- some way to get queues named (aliasing for queues)
Any comments, feedback, questions? Should I just stick an enhancement in
Bugzilla, for whenever there's time and/or a sponsor?
Best regards,
Radu
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
