On Tue, 22 Nov 2016, Rich Megginson wrote:
On 11/22/2016 06:47 AM, David Lang wrote:
On Tue, 22 Nov 2016, Rainer Gerhards wrote:
if you are interested in a very lightweight log shipper for files, I
would appreciate your feedback on this:
http://blog.gerhards.net/2016/11/would-creating-simple-linux-log-file.html
posted to the blog, but repeating here:
I have not found a good tool yet (I've written or seen written a couple
over the years)
The problem tends to be that there is not really such a thing as 'no
additional processing needed', any one use requires a tiny fraction of the
capabilities that rsyslog offers, but each setup requires a different
combination of capabilities.
What you would have in this case (and this is the use case of other
lightweight collectors such as filebeat and fluentbit) is the collector agent
running on the node would do "practically" no log processing, and instead
ship the logs to a cluster of "heavyweight" rsyslogs doing additional
processing. This allows you to
* make the node as small and fast as possible
* use all of the capabilities and plugins of rsyslog to do your additional
processing
* scale up the additional processing
my problem has been that I find you tend to loose metadata that you need to have
to properly do the additional processing (exactly what directory did this file
come from, what is the timezone on the sending machine, etc)
just look at all the capabilities that imfile has been growing over the
last few years.
I suspect that a stripped down compile of rsyslog (no input modules other
than imfile, especially no imjournal, etc) would end up being competitive
to just about any special-purpose program.
+1 - rsyslog is already much more scalable than most other log collectors.
Why "especially no imjournal"?
As per a discussion here a couple months ago, adding imjournal more than doubles
the memory footprint of a running rsyslog instance, even if it's not used.
IMHO, The biggest problem with using rsyslog to do this is the same problem
we have with using rsyslog to create /dev/log in containers, the fact that
the config is fixed at startup time.
Can you explain more about what you mean by this?
rsyslog can create additional /dev/log equivalent sockets, and as such, you
could create them in each container so that there is no need for a copy of
rsyslog in each container.
The problem is that since the config of rsyslog if fixed, the directories the
containers are going to be in would all have to exist at the time rsyslog
starts. And those directories could not be removed until rsyslog stops (and
closes the files it has open)
In the world where you spin up containers as load changes, and shift them from
machine to machine to balance the load, this is a problem.
Similarly with reading files, you frequently want to start a new process that
will create files, and not have to change the rsyslog config and restart it
(affecting the processing of other logs) just to start grabbing a new file.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
