@orangepeel beef, In earlier discussions others have mentioned using logstash as a second indexer, I chose to not include it because of performance hits that were mentioned. Currently I'm only in a test phase of establishing my syslog server with this solution. I have only 2 hosts logging to it. My windows laptop, a windows server and the syslog server's own logs. Just those 3 and refreshing in kibana causes some major performance concerns. If I actually added all the other hosts that I plan to I think it would crash altogether.
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Orangepeel Beef Sent: Wednesday, May 14, 2014 3:19 PM To: rsyslog-users Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server There are a ton of headaches associated with directly logging to elasticsearch as well. How do you reindex if an index crashes if you are not storing your logs somewhere else as an intermediary? ES crashes indexes if it runs out of memory, or disk space, and they crash hard. I've rebuilt indexes many many times already. What happens when you have a large burst of traffic and elasticsearch can't handle it? rsyslog can handle a very large amount of throughput, and writing to files it won't lose anything, but writing to es, it can. How do you pass data to Simple event correlator and then into elasticsearch? pipe it out, and then back into rsyslog? no thanks. How do you tag different file types if you are sending direct to ES? each one of my different logtypes has patterns and filters setup to parse data out of them that rely on the type being set appropriately. How do you grok parse fields if you are going direct to ES? Logstash does that bit, and you're bypassing it here. I work in network security and can't lose pretty much *any* logs. Logging to file bypasses all of these issues, and the logstash file input maintains a sincedb state of file positioning and can index at its leisure, even if logstash is stopped and restarted, it will pick up from where it left off. Plus we have requirements to maintain the logs for 6+ months, but we do not need to maintain 6 month elasticsearch searchable data. I keep 3 days of uncompressed raw logs for easy indexing / reindexing, and everything older than that is bzipped, backed up, and stored. Sure you could use elasticsearch-knapsack to export/backup your ES data, but it's far easier to just maintain the raw logs. But hey, to each their own. On Wed, May 7, 2014 at 12:43 PM, David Lang <[email protected]> wrote: > at my old job we had ossec configured to send to rsyslog > > personally I really dislike the 'write to a file and then scrape it > with another program' approach to logs > > Yes, it handles cases where your logserver is down, but you should > have HA so that's a very rare case. > > But it causes a bunch of headaches > > 1. a lot more disk I/O > > 2. polling to check if the file has changed > > 3. headaches if the files roll too fast > > 4. problems deciding when you can delete the files > > It's just so much easier to pass the data directly to rsyslog and let > it deal with everything :-) > > David Lang > > > > On Wed, 7 May 2014, Josh Bitto wrote: > > Date: Wed, 7 May 2014 09:44:43 -0700 >> >> From: Josh Bitto <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana >> server >> >> Hello Everyone and Good Morning! >> >> I have a new question for you all. Does anyone have this current >> setup with an OSSEC server as well? I'm wondering which would be the >> better option to do. Just create an imfile for Rsyslog to monitor the >> logs from OSSEC or forward them to rsyslog. I'm curious to find out >> if anyone else has this implemented too! >> >> >> Josh >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
