I think this actually could be a good addition for some use cases. Pulling internal stats, error reports and regular logs is definitely something of use. I thought about a similar capability some time ago (albeit in a different context, but the core idea was the same).
The problem is that it would require quite some changes, even where you would not necessary expect them (e.g. the queue subsystem would need to have some serious refactoring, as it is based on the push model, too). In any case, this work would need a sponsor. I think it's something that does require two to three month of work at minimum. If a sponsor comes up, I would definitely like to do that work - without one, there are many more more-frequently-requested things (among others, I would really like to rewrite the queue and batching code with all the exprience we gained in the past years - would definitely be very useful. This could also lay the foundation for a pull model from the queue's PoV). With that said: it probably is a good idea to add it to the bug tracker, even though I think it will rest there with other enhancements ;) But it may attract a sponsor... Rainer On Tue, Oct 8, 2013 at 10:17 AM, Radu Gheorghe <[email protected]>wrote: > Hello, > > I know it's a long tail of "interesting features" that's waiting to be > implemented, and adding yet another one might be... simply too much. But > this one popped in my head as being reaaally nice for more than an year - > even though I don't particularly need it - and I've never shared it. Which > also feels wrong. So here I am, sharing it :) > > The idea is simple: expose a REST API that allows external applications to > pull data from queues. Something like this: > > curl 'rsyslog-host:5514/queue1' > #and the answer would be something like > { > "took": 75, > "items": [ > { > "msg": "hello world!", > "hostname": "another-host", > .......other properties..... > }, > { > [another message would go here. X messages in total, for > queue.dequebatchsize=X] > } > ] > } > > Whatever gets into those JSON items would be decided by the templates, and > the module implementing this REST API would be an output module, holding > those items in its action queue. > > Maybe the [high-level] implementation details above aren't so great. Please > comment if you think there are better ways of doing it. Still, I think the > idea is valueable: provide a [simple] way for transmitting messages via a > pull mechanism rather than push. > > A pull mechanism has its advantages (and disadvantages, of course) over > push: > - whatever pulls those messages won't ever be overwhelmed, as it can only > pull data as fast as it can process it. I had Elasticsearch > rivers< > http://www.elasticsearch.org/guide/en/elasticsearch/rivers/current/river.html > >in > mind, but really, any pluggable system should be able to pull data > from > rsyslog and process it > - some use-cases require firewalls to be set up in such a way that pulling > is more convenient > - when you need to centralize logs, a setup like this seems more and more > popular (see how Logstash recommends > Redis<http://logstash.net/docs/1.2.1/tutorials/getting-started-centralized > >as > buffer): > *[log producer]-->[buffer]-->[data store]* > And having rsyslog act like a dedicated buffer will allow anyone to write > plugins and services to pull those messages and process them, without being > limited to syslog protocols or to writing rsyslog modules. This should open > rsyslog (and the syslog world in general) to get better integration with > lots of data processing and monitoring tools. For example, one can stick > the impstats logs in such a queue and write a Nagios plugin to monitor > rsyslog metrics via this REST API. > > It feels like most of the needed parts are already there: templates, > queues, debatchsize, discarding when the queue is full (even by severity). > Maybe only two missing parts: > - the output module for this REST API > - some way to get queues named (aliasing for queues) > > Any comments, feedback, questions? Should I just stick an enhancement in > Bugzilla, for whenever there's time and/or a sponsor? > > Best regards, > Radu > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
