On 11/22/2016 06:47 AM, David Lang wrote:
On Tue, 22 Nov 2016, Rainer Gerhards wrote:
if you are interested in a very lightweight log shipper for files, I
would appreciate your feedback on this:
http://blog.gerhards.net/2016/11/would-creating-simple-linux-log-file.html
posted to the blog, but repeating here:
I have not found a good tool yet (I've written or seen written a
couple over the years)
The problem tends to be that there is not really such a thing as 'no
additional processing needed', any one use requires a tiny fraction of
the capabilities that rsyslog offers, but each setup requires a
different combination of capabilities.
What you would have in this case (and this is the use case of other
lightweight collectors such as filebeat and fluentbit) is the collector
agent running on the node would do "practically" no log processing, and
instead ship the logs to a cluster of "heavyweight" rsyslogs doing
additional processing. This allows you to
* make the node as small and fast as possible
* use all of the capabilities and plugins of rsyslog to do your
additional processing
* scale up the additional processing
just look at all the capabilities that imfile has been growing over
the last few years.
I suspect that a stripped down compile of rsyslog (no input modules
other than imfile, especially no imjournal, etc) would end up being
competitive to just about any special-purpose program.
+1 - rsyslog is already much more scalable than most other log
collectors. Why "especially no imjournal"?
IMHO, The biggest problem with using rsyslog to do this is the same
problem we have with using rsyslog to create /dev/log in containers,
the fact that the config is fixed at startup time.
Can you explain more about what you mean by this?
Having a command socket that rsyslog listened to that would let you
add/remove inputs (files or unix sockets), but didn't allow you to
change anything else in the config would let you easily tell rsyslog
to start watching a new container or file as needed, and then stop
watching so that it doesn't prevent the container or directory from
going away when the app/container is removed.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
