I understand you need to keep the architecture scaled down as a requirement, but wanted to comment on this for anyone searching this thread without the same requirement. I don't consider indexing a performance gate...at least not one you can't design around if you put middleware to good use.
I very much want to have rsylog front-ends, and even relays (writing to files + sending to my middleware), but logstash is fine for indexing. I can configure it to use any number of threads, and run any number of instances across a boatload of VMs to easily scale. I'm doing that now in fact. My biggest annoyance is having some environments (which I'm happy with) where the mentioned front-ends are already rsyslog (I built those) and others I inherited which use logstash to ingest the initial traffic...and get overloaded/crash/etc. Sure I can scale that out too, and logstash gets better all the time, but rsyslog is lightyears ahead in speed and stability. It's a better fit for the task, based on personal experience. -----Original Message----- From: Josh Bitto <[email protected]> Reply-To: rsyslog-users <[email protected]> Date: Wednesday, May 14, 2014 at 6:39 PM To: rsyslog-users <[email protected]> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server >@orangepeel beef, > >In earlier discussions others have mentioned using logstash as a second >indexer, I chose to not include it because of performance hits that were >mentioned. Currently I'm only in a test phase of establishing my syslog >server with this solution. I have only 2 hosts logging to it. My windows >laptop, a windows server and the syslog server's own logs. Just those 3 >and refreshing in kibana causes some major performance concerns. If I >actually added all the other hosts that I plan to I think it would crash >altogether. > > > >-----Original Message----- >From: [email protected] >[mailto:[email protected]] On Behalf Of Orangepeel Beef >Sent: Wednesday, May 14, 2014 3:19 PM >To: rsyslog-users >Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server > >There are a ton of headaches associated with directly logging to >elasticsearch as well. > >How do you reindex if an index crashes if you are not storing your logs >somewhere else as an intermediary? ES crashes indexes if it runs out of >memory, or disk space, and they crash hard. I've rebuilt indexes many >many times already. > >What happens when you have a large burst of traffic and elasticsearch >can't handle it? rsyslog can handle a very large amount of throughput, >and writing to files it won't lose anything, but writing to es, it can. > >How do you pass data to Simple event correlator and then into >elasticsearch? pipe it out, and then back into rsyslog? no thanks. > >How do you tag different file types if you are sending direct to ES? >each one of my different logtypes has patterns and filters setup to parse >data out of them that rely on the type being set appropriately. > >How do you grok parse fields if you are going direct to ES? Logstash >does that bit, and you're bypassing it here. > >I work in network security and can't lose pretty much *any* logs. >Logging to file bypasses all of these issues, and the logstash file input >maintains a sincedb state of file positioning and can index at its >leisure, even if logstash is stopped and restarted, it will pick up from >where it left off. >Plus we have requirements to maintain the logs for 6+ months, but we do >not need to maintain 6 month elasticsearch searchable data. > >I keep 3 days of uncompressed raw logs for easy indexing / reindexing, and >everything older than that is bzipped, backed up, and stored. Sure you >could use elasticsearch-knapsack to export/backup your ES data, but it's >far easier to just maintain the raw logs. > >But hey, to each their own. > > > > > > >On Wed, May 7, 2014 at 12:43 PM, David Lang <[email protected]> wrote: > >> at my old job we had ossec configured to send to rsyslog >> >> personally I really dislike the 'write to a file and then scrape it >> with another program' approach to logs >> >> Yes, it handles cases where your logserver is down, but you should >> have HA so that's a very rare case. >> >> But it causes a bunch of headaches >> >> 1. a lot more disk I/O >> >> 2. polling to check if the file has changed >> >> 3. headaches if the files roll too fast >> >> 4. problems deciding when you can delete the files >> >> It's just so much easier to pass the data directly to rsyslog and let >> it deal with everything :-) >> >> David Lang >> >> >> >> On Wed, 7 May 2014, Josh Bitto wrote: >> >> Date: Wed, 7 May 2014 09:44:43 -0700 >>> >>> From: Josh Bitto <[email protected]> >>> Reply-To: rsyslog-users <[email protected]> >>> To: rsyslog-users <[email protected]> >>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana >>> server >>> >>> Hello Everyone and Good Morning! >>> >>> I have a new question for you all. Does anyone have this current >>> setup with an OSSEC server as well? I'm wondering which would be the >>> better option to do. Just create an imfile for Rsyslog to monitor the >>> logs from OSSEC or forward them to rsyslog. I'm curious to find out >>> if anyone else has this implemented too! >>> >>> >>> Josh >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >_______________________________________________ >rsyslog mailing list >http://lists.adiscon.net/mailman/listinfo/rsyslog >http://www.rsyslog.com/professional-services/ >What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE >THAT. >_______________________________________________ >rsyslog mailing list >http://lists.adiscon.net/mailman/listinfo/rsyslog >http://www.rsyslog.com/professional-services/ >What's up with rsyslog? Follow https://twitter.com/rgerhards >NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
