On Tue, 22 Nov 2016, [email protected] wrote:
http://blog.gerhards.net/2016/11/would-creating-simple-linux-log-file.html
IMHO: the "issue" we're having with rsyslog is understanding the code rather
than performance. Why you'll invest time doing a "faster reader" if you
already have a "fast-enough processor"? Has anyone complained about speed?
What would be the benefit from such development? Will it increase reading
speed by 2x? 4x? 10x?
I suggest you taking this approach:
* Read file line by line (consider also it handles multiline)
* Measure speed (lines/sec, MB/sec, chars/sec...
* Read the same file with imfile with similar reading configuration
* Measure speed (lines/sec, MB/sec, chars/sec...
* Compare both times and think what you can do to make it faster :P
I have not found a good tool yet (I've written or seen written a couple
over the years)
What are the issues with imfile? I don't know when rsyslog development
started, but perhaps time for refactoring has come.
well, the last time I had to do this, the problem was that the company didn't
want to spend the money for rsyslog on Windows. The time before that it was that
rsyslog wasn't available on AIX
The time before that I created readmode in imfile to address some of the issues.
:-)
That said, here is where I still see issues just using imfile
1. it's hard to extract the proper metadata out of imfile and use it in the
resulting log.
imfile was designed for the case where the file has no meaningful structure. But
if there is structure, it's hard for imfile to take advantage of it (requiring
custom templates to replicate the existing log formats) having the ability to
use mmnormalize to populate the default properties, or an easy way to override
the default properties in an output would help a lot
2. you can't add a file to be monitored without having to modify the rsyslog
config and do a full stop/start, which impacts other uses
I suspect that a stripped down compile of rsyslog (no input modules other
than imfile, especially no imjournal, etc) would end up being competitive
to just about any special-purpose program.
I don't know if I understood properly: is there such a thing as
--disable-imjournal? will this make rsyslog faster?
it would make it a lot smaller
Is there any place where I can see "everything than can be disabled in order
to make rsyslog even faster"?
it's less a matter of speed than it is size, rsyslog is getting larger with all
the libraries it's using, which makes it arguably less appropriate for such a
simple task.
The default queue size is also probably not appropriate for this simple
use-case. Why read so much from the file and put it in RAM?
./configure --help shows these options. I'm wondering if having a minimal build
that created a different binary, used a different config and stripped everything
down would be useful
IMHO, The biggest problem with using rsyslog to do this is the same problem
we have with using rsyslog to create /dev/log in containers, the fact that
the config is fixed at startup time.
That's somehow why I was asking about imhiredis.
I don't know yet how rsyslog works internally, but I'll love being able to
reload/add/remove configuration/pipelines during execution.
Rsyslog does a lot of things as it starts up, this includes loading modules,
which each provide code for interpreting the config file, and then running the
parsed configuration through an optimizer.
This makes a pretty significant difference (a lot of the config statements that
make a lot of sense to a person are slow to interpret, and things that are fast
to process are harder to read)
so adding/removing large chunks of config dynamically is probably not a
reasonable thing to do.
Adding/removing inputs where the rest of the processing/config does not change
is much less disruptive.
Having a command socket that rsyslog listened to that would let you
add/remove inputs (files or unix sockets), but didn't allow you to change
anything else in the config would let you easily tell rsyslog to start
watching a new container or file as needed, and then stop watching so that
it doesn't prevent the container or directory from going away when the
app/container is removed.
It's all this because rsyslogs polls for new files/dir changes or something
like that?
yes, rsyslog maintains an open filehandle (/dev/log) for unix sockets, and holds
the directory open to see file changes. These prevent removing/unmounting these
directories.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
