It's just my opinion, but I think this is mostly a matter of perspective and what you have control over.
If logs "appear" in files and you have no real control over how they got there (such as applications logging directly to files rather than to syslog), and your goal is just to get them into Elastic and Kibana, then I would probably pick FileBeat reading the files over rsyslog using imfile to read the files. However, if the process that put those logs in the files is in fact rsyslog and you can control it's configuration, I think it makes more sense to use rsyslog to ship to Elastic (either directly or via intermediary kafka). As you suggested, there's no point putting them in files on disk only to slurp them back up again with another tool to ship them somewhere else. If you already have built a reliable syslog collection infrastructure (so you've already addressed UDP vs TCP, disk-assisted queueing, pstats collection, buffer sizing, etc.) and do additional work such as log normalization, JSON encoding, additional metadata collection, etc. in that pipeline, then that's a lot of functionality you would have to reproduce (and keep in sync over time) in the FileBeat pipeline. Also consider the expected volume and relative importance of the new data feed. If your current rsyslog pipeline is handling "very important" data, and you are anticipating the new data stream to be massive and relatively low value, then it might make sense to keep it entirely separate (transported via FileBeat). We have a "critical path" rsyslog pipeline for feeding log analysis applications, and have also separately used FileBeat to collect container logs from pods in Kubernetes to keep them out of the critical path and just ship them to Elastic & Kibana for troubleshooting use, for example. Best regards, - Dave Caplinger On Feb 28, 2019, at 11:23 AM, Singh, Radesh <[email protected]<mailto:[email protected]>> wrote: Hello, First of all, not trying to start a war ( I promise! ), just want to get some fresh perspective on something. We have several rsyslog servers where the server is configured to forward logs (received on an IP port) to another rsyslog host, or some other destination. Recently I started getting requests to install filebeat and use filebeat to watch a log file for the purposes of sending to ELK, NiFi, etc. Since we've always used syslog to receive logs and forward, the question I've got is ... why add another tool into the mix? Reading a very little bit, this post on elastic.co<http://elastic.co>: https://discuss.elastic.co/t/filebeat-vs-syslog/50547 I get the impression that filebeat may offer some more reliability; however, I'm not convinced. Wanted to throw the question out as I'm sure someone has considered this situation and may have some thoughts. Thanks! Radesh _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. Confidentiality Notice: The content of this communication, along with any attachments, is covered by federal and state law governing electronic communications and may contain confidential and legally privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, use or copying of the information contained herein is strictly prohibited. If you have received this communication in error, please immediately contact us by telephone at 402.361.3000 or e-mail [email protected]. Copyright 2000-2018 NTT Security (US) Inc., a wholly-owned subsidiary of NTT Group. All rights reserved. NTT Security is a trademark of NTT Security GMBH. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
