2013/10/8 David Lang <[email protected]> > The big problem with this idea is that in general messages do not stay in > the queues for very long. They are there only until they can get delivered > to their destinations. > > If you were to have rsyslog stop delivering the messages to a destination > and then hang on to them all, how would rsyslog ever know if all the > destinations had completed their polling? >
Ah, I think there's a misunderstanding here. I'm not suggesting to give access to any action's queue. I'm suggesting to have a separate queue for the REST API. For example, rsyslog is listening to /dev/log, and instead of forwarding those messages via TCP, put them in the action queue and wait for something to pull those messages from the queue. When once asks for a batch of messages, they get "consumed". > > It is easy enough to have rsyslog deliver messages to a queue of your > choice (may I suggest that you look at either Zero MQ or Rabbit MQ which > are already supported by rsyslog), and then you can deal with them at your > convienience. > Right, thanks for mentioning. This will work as well. Although having the REST API directly in rsyslog will eliminate the need of another moving piece. > > > There's also the problem that giving outside programs access to the > rsyslog internals would be fragile, it wouldn't work with any other syslog > daemon (destroying the value of standardization), and it would probably not > continue to work even with rsyslog in the face of updates or even config > changes. > Ah, but this has nothing to do with syslog standards. Think of it as a different output, like omelasticsearch. Only instead of pushing logs, rsyslog will store them so they can get pulled. > > As for the logstash recommendations, I can tell you that the idea of > putting all your logs through a single chain of processing the way they do > doesn't scale that well. At really high log volumes, you need to be able to > split your processing not only across different servers, but across > different farms of servers. > The "ASCII diagram" was just a sketch of the flow, I didn't include scaling in there. With Logstash in particular, you can have multiple "receivers" across multiple servers, then if you use Redis as a buffer you can cluster that, and you can have multiple Logstash "indexers" to send logs to Elasticsearch, which can be clustered, too. Maybe my drawing was confusing, sorry for that. I wanted to show the chain of processing in terms of technology, not in terms of scale. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
