In your flow below, logstash could become a bottleneck...you'd design around that with load balancing.
The reason people say logstash is a better indexer is how closely related elasticsearch/logstash/kibana now are -- the embedded client in logstash is likely to be updated/optimized quicker, if you always follow the latest logstash+elasticsearch releases (no reason not to, I've seen quite the opposite in fact with a lot of bug fixes from ls 1.2->1.3->1.4). That said, I can't comment on a real comparison since I haven't ran both side by side in the indexer role... I'm sure others can. FWIW, my current setup looks like (sorry if the arrows are confusing, trying to be pedantic and imply way too much information with ASCII): clients -> vip -> [rsyslogs|logstashes] -> vip -> rabbitmq spokes <-(federation)- rabbitmq hubs <- vip <- logstash indexers -> vip -> elasticsearch masters -> elasticsearch data <- elasticsearch search <- kibana <- users I'd like to fully eliminate logstash on the left, since that's where it's given me problems... otherwise this works. Federation (AMQP) does well over long-distance/high latency links, I can have any number of indexers digesting messages as fast as possible, and even if all indexers are down I don't loose anything. -----Original Message----- From: Josh Bitto <[email protected]> Reply-To: rsyslog-users <[email protected]> Date: Friday, May 16, 2014 at 5:33 PM To: rsyslog-users <[email protected]> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server >A quick question on this. > >Currently the way my architecture looks like is this: >Remote hosts->central syslog server (where logs are processed in this >order) rsyslog + omelasticsearch->elasticsearch->kibana3 > >What would the advantages and disadvantages be of doing the following: >Remote hosts->central syslog server where rsyslog + >omelasticsearch->logstash->elasticsearch->kibana3 > >The reason I ask is yesterday I talked with some peeps in the logstash >irc channel (on a different issue) and they pretty much said that >logstash was a much better indexer than rsyslog. I haven't used logstash >to really make a comparison, but my preference is rsyslog due to the fact >that the scripting language is easier for me to learn and understand. >Currently I'm experiencing on this test environment where elasticsearch >wants to use as much memory as possible and I'm only testing 3 hosts to >it right now with very minimal logging. > > > > > >-----Original Message----- >From: [email protected] >[mailto:[email protected]] On Behalf Of Mike Hoskins >(michoski) >Sent: Friday, May 16, 2014 2:11 PM >To: rsyslog-users >Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server > >I understand you need to keep the architecture scaled down as a >requirement, but wanted to comment on this for anyone searching this >thread without the same requirement. I don't consider indexing a >performance gate...at least not one you can't design around if you put >middleware to good use. > >I very much want to have rsylog front-ends, and even relays (writing to >files + sending to my middleware), but logstash is fine for indexing. I >can configure it to use any number of threads, and run any number of >instances across a boatload of VMs to easily scale. I'm doing that now >in fact. > >My biggest annoyance is having some environments (which I'm happy with) >where the mentioned front-ends are already rsyslog (I built those) and >others I inherited which use logstash to ingest the initial traffic...and >get overloaded/crash/etc. Sure I can scale that out too, and logstash >gets better all the time, but rsyslog is lightyears ahead in speed and >stability. It's a better fit for the task, based on personal experience. > >-----Original Message----- >From: Josh Bitto <[email protected]> >Reply-To: rsyslog-users <[email protected]> >Date: Wednesday, May 14, 2014 at 6:39 PM >To: rsyslog-users <[email protected]> >Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server > >>@orangepeel beef, >> >>In earlier discussions others have mentioned using logstash as a second >>indexer, I chose to not include it because of performance hits that >>were mentioned. Currently I'm only in a test phase of establishing my >>syslog server with this solution. I have only 2 hosts logging to it. My >>windows laptop, a windows server and the syslog server's own logs. Just >>those 3 and refreshing in kibana causes some major performance >>concerns. If I actually added all the other hosts that I plan to I >>think it would crash altogether. >> >> >> >>-----Original Message----- >>From: [email protected] >>[mailto:[email protected]] On Behalf Of Orangepeel Beef >>Sent: Wednesday, May 14, 2014 3:19 PM >>To: rsyslog-users >>Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server >> >>There are a ton of headaches associated with directly logging to >>elasticsearch as well. >> >>How do you reindex if an index crashes if you are not storing your logs >>somewhere else as an intermediary? ES crashes indexes if it runs out >>of memory, or disk space, and they crash hard. I've rebuilt indexes >>many many times already. >> >>What happens when you have a large burst of traffic and elasticsearch >>can't handle it? rsyslog can handle a very large amount of throughput, >>and writing to files it won't lose anything, but writing to es, it can. >> >>How do you pass data to Simple event correlator and then into >>elasticsearch? pipe it out, and then back into rsyslog? no thanks. >> >>How do you tag different file types if you are sending direct to ES? >>each one of my different logtypes has patterns and filters setup to >>parse data out of them that rely on the type being set appropriately. >> >>How do you grok parse fields if you are going direct to ES? Logstash >>does that bit, and you're bypassing it here. >> >>I work in network security and can't lose pretty much *any* logs. >>Logging to file bypasses all of these issues, and the logstash file >>input maintains a sincedb state of file positioning and can index at >>its leisure, even if logstash is stopped and restarted, it will pick up >>from where it left off. >>Plus we have requirements to maintain the logs for 6+ months, but we do >>not need to maintain 6 month elasticsearch searchable data. >> >>I keep 3 days of uncompressed raw logs for easy indexing / reindexing, >>and >>everything older than that is bzipped, backed up, and stored. Sure you >>could use elasticsearch-knapsack to export/backup your ES data, but >>it's far easier to just maintain the raw logs. >> >>But hey, to each their own. >> >> >> >> >> >> >>On Wed, May 7, 2014 at 12:43 PM, David Lang <[email protected]> wrote: >> >>> at my old job we had ossec configured to send to rsyslog >>> >>> personally I really dislike the 'write to a file and then scrape it >>> with another program' approach to logs >>> >>> Yes, it handles cases where your logserver is down, but you should >>> have HA so that's a very rare case. >>> >>> But it causes a bunch of headaches >>> >>> 1. a lot more disk I/O >>> >>> 2. polling to check if the file has changed >>> >>> 3. headaches if the files roll too fast >>> >>> 4. problems deciding when you can delete the files >>> >>> It's just so much easier to pass the data directly to rsyslog and let >>> it deal with everything :-) >>> >>> David Lang >>> >>> >>> >>> On Wed, 7 May 2014, Josh Bitto wrote: >>> >>> Date: Wed, 7 May 2014 09:44:43 -0700 >>>> >>>> From: Josh Bitto <[email protected]> >>>> Reply-To: rsyslog-users <[email protected]> >>>> To: rsyslog-users <[email protected]> >>>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana >>>> server >>>> >>>> Hello Everyone and Good Morning! >>>> >>>> I have a new question for you all. Does anyone have this current >>>> setup with an OSSEC server as well? I'm wondering which would be the >>>> better option to do. Just create an imfile for Rsyslog to monitor >>>> the logs from OSSEC or forward them to rsyslog. I'm curious to find >>>> out if anyone else has this implemented too! >>>> >>>> >>>> Josh >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> you DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you DON'T LIKE THAT. >>> >>_______________________________________________ >>rsyslog mailing list >>http://lists.adiscon.net/mailman/listinfo/rsyslog >>http://www.rsyslog.com/professional-services/ >>What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >>This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >>beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>LIKE THAT. >>_______________________________________________ >>rsyslog mailing list >>http://lists.adiscon.net/mailman/listinfo/rsyslog >>http://www.rsyslog.com/professional-services/ >>What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >>This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >>beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>LIKE THAT. > >_______________________________________________ >rsyslog mailing list >http://lists.adiscon.net/mailman/listinfo/rsyslog >http://www.rsyslog.com/professional-services/ >What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE >THAT. >_______________________________________________ >rsyslog mailing list >http://lists.adiscon.net/mailman/listinfo/rsyslog >http://www.rsyslog.com/professional-services/ >What's up with rsyslog? Follow https://twitter.com/rgerhards >NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
