On Fri, 7 Dec 2012 10:34:25 -0800 (PST) David Lang <[email protected]> wrote:
> On Fri, 7 Dec 2012, Ben Bradley wrote: > > I'd like to store the logs on a central server running > > logstash/ElasticSearch so they can be searched and monitored using Kibana. > > With rsyslog sending the logs over the network to a logstash server. I > > don't want to run logstash as the log "sender" on each server, I'd prefer > > to keep the servers (log "clients") as lean and simple possible. So that > > means either using syslog, syslog-ng or the one I'm testing now, rsyslog. > > > > 1) Should I have rsyslog sending to logstash over the network? Or should I > > be > > running another rsyslog on the collector server, which then sends to > > logstash > > for processing? > > This is up to you, there are advantages in each direction. > > Using rsyslog for all network transport and having it deliver locally to > logstash/elasticsearch/other for processing means that you you can take > advantage of all rsyslog features for your transport. > > In a centralized environment your traffic volumes can be high, rsyslog can > handle very high traffic levels, can your other software? > > If you are really comforatable with logstash, you may want to eliminate the > need > to run one more daemon, but when you hire new people and hand the system > over, > should they need to be as comfortable with logstash? They will have to be > comforatable with rsyslog in any case. At that point which is the 'extra' > thing > to deal with, rsyslog or logstash? This is very interesting and I agree. The simpler the better. The thing I like about logstash is that it outputs to ElasticSearch by default. Replacing logstash and having rsyslog save to ElasticSearch seems a bit more complicated and experimental. We are starting off small with this but depending on load I may way do as you suggest and have rsyslog as the collector to deal with high loads. > > For Apache, I would like to have separate vhost log files on the web > > server, in addition to these logs being sent to a remote log collector. > > > > I've tested rsyslog using the imfile module to watch each Apache log files, > > but this means I have to hard-code each vhost log file into my > > rsyslog.conf. This is not ideal as people will invariably forget when they > > add/remove sites on the server. > > > > 2) What's the best way to log to both vhost-specific log files on the web > > server and to send these logs over the network, without using imfile and > > manually watching tens of individual log files? > > Get Apache to log to rsyslog, then have rsyslog split the log to both a > > file and over the network to logstash? > > Are there big performance implications for logging both locally and over > > the network? > > > > I could change my Apache config to log to a single access/error log for all > > vhosts, then watch these main log files with imfile. So long as rsyslog is > > then able to produce vhost-specific log files somewhere on the web server > > machine. > > it depends on how you format the log file. If you have the logfile start with > the vhost name, then rsyslog can easily produce per-host files (look in the > rsyslog documentation for the dynafile templates. That's interesting. I will definitely take a look at that. > another approach you can do is have apache log to a local named pipe and have > a > process listen on that named pipe and tagging/reformatting the log file and > pass > it to your syslog server. > Do you mean something like this? http://serverfault.com/questions/385414/apache-httpd-send-error-logs-to-syslog-and-local-disk-without-touching-etc-sy Using a script like this? http://www.oreillynet.com/pub/a/sysadmin/2006/10/12/httpd-syslog.html Thanks for the info! _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
