David, I understand what you and orangebeef are trying to say and I have taken it into serious consideration. The only thing that becomes an issues is actual resources to have that type of infrastructure. That is why one syslog server is the only practical solution that our company has right now. If we had the budget to do more I would. So my only option is one server with all logs going to it. Personally I like the idea of having 2 or 3 syslog servers with splunk on all of them processing 500mb a day for free. :)
I'm going to tackle this issue again today and see what I can come up with. -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of David Lang Sent: Wednesday, May 14, 2014 10:51 PM To: rsyslog-users Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server I actually like having rsyslog send the logs directly to ES, but I agree that ES should not be the only place the logs go to. As Orangepeel Beef notes, there are many other destinations that you want your logs to go to. I like to keep a feed of the logs as a plain, combined feed (so I can easily see what's going on across multiple machines without having to go into all the different directories) Sending a copy of the logs to Simple Event correlator is extremely handy and then you really want the logs from your core server to exist elsewhere so that when you have problems with it, you can figure out what's going wrong so I like to have a rule if $fromhost-ip == "127.0.0.1" then { <set some vars> @log-relay;JSON_format } else { send to local combined file send to file split by server send to file split by programname send to SEC (split by programname) send to ES (or other search tool) etc } When you get to the point of making everything redundant, and/or splitting the workload across servers for different functions, having all your logs, including from your log servers, relayed through your relay boxes makes things "just work" David Lang On Wed, 14 May 2014, Orangepeel Beef wrote: > There are a ton of headaches associated with directly logging to > elasticsearch as well. > > How do you reindex if an index crashes if you are not storing your > logs somewhere else as an intermediary? ES crashes indexes if it runs > out of memory, or disk space, and they crash hard. I've rebuilt > indexes many many times already. > > What happens when you have a large burst of traffic and elasticsearch > can't handle it? rsyslog can handle a very large amount of > throughput, and writing to files it won't lose anything, but writing to es, > it can. > > How do you pass data to Simple event correlator and then into > elasticsearch? pipe it out, and then back into rsyslog? no thanks. > > How do you tag different file types if you are sending direct to ES? > each one of my different logtypes has patterns and filters setup to > parse data out of them that rely on the type being set appropriately. > > How do you grok parse fields if you are going direct to ES? Logstash > does that bit, and you're bypassing it here. > > I work in network security and can't lose pretty much *any* logs. > Logging to file bypasses all of these issues, and the logstash file > input maintains a sincedb state of file positioning and can index at > its leisure, even if logstash is stopped and restarted, it will pick up from > where it left off. > Plus we have requirements to maintain the logs for 6+ months, but we > do not need to maintain 6 month elasticsearch searchable data. > > I keep 3 days of uncompressed raw logs for easy indexing / reindexing, and > everything older than that is bzipped, backed up, and stored. Sure you > could use elasticsearch-knapsack to export/backup your ES data, but > it's far easier to just maintain the raw logs. > > But hey, to each their own. > > > > > > > On Wed, May 7, 2014 at 12:43 PM, David Lang <[email protected]> wrote: > >> at my old job we had ossec configured to send to rsyslog >> >> personally I really dislike the 'write to a file and then scrape it >> with another program' approach to logs >> >> Yes, it handles cases where your logserver is down, but you should >> have HA so that's a very rare case. >> >> But it causes a bunch of headaches >> >> 1. a lot more disk I/O >> >> 2. polling to check if the file has changed >> >> 3. headaches if the files roll too fast >> >> 4. problems deciding when you can delete the files >> >> It's just so much easier to pass the data directly to rsyslog and let >> it deal with everything :-) >> >> David Lang >> >> >> >> On Wed, 7 May 2014, Josh Bitto wrote: >> >> Date: Wed, 7 May 2014 09:44:43 -0700 >>> >>> From: Josh Bitto <[email protected]> >>> Reply-To: rsyslog-users <[email protected]> >>> To: rsyslog-users <[email protected]> >>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana >>> server >>> >>> Hello Everyone and Good Morning! >>> >>> I have a new question for you all. Does anyone have this current >>> setup with an OSSEC server as well? I'm wondering which would be the >>> better option to do. Just create an imfile for Rsyslog to monitor >>> the logs from OSSEC or forward them to rsyslog. I'm curious to find >>> out if anyone else has this implemented too! >>> >>> >>> Josh >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
