2014-07-13 3:31 GMT+08:00 David Lang <[email protected]>:
> On Sat, 12 Jul 2014, Justin Lee wrote:
>
>> Hello List,
>>
>> I noticed that local messages are passed by socket rather than
>> directly written to log files. The evidence is as follows:
>
>
> Yes, this is how syslog works.
>
>
>> Since log mechanism is an important infrastructure of many programs,
>> it is supposed to be as fast as possible. So do we have some
>> configurations or settings in rsyslog which make local messages
>> written directly into log files without passing through socket?
>
>
> actually, writing to disk can be slower than writing to a socket, so it's
> not clear that it's a performance win to write to disk directly.
I think that writing to disk is supposed to be the final step of
rsyslogd in processing local messages. After all, we couldn't see the
messages in log files if rsyslogd didn't write to disk at all, right?
But in addition to disk writing, use of socket by syslog to transfer
local log messages introduces extra CPU time or memory to do the
following things:
1. To handle socket related stuff, such as copying log messages in and
out of socket buffer. Buffer copying introduces overhead.
2. We need a thread or daemon process (rsyslogd) to listen to and
receive log messages from the socket. It increases the memory
footprint of our log infrastructure because of the daemon process
itself.
3. At least one context switch is required to transfer a log message
from user program to rsyslog daemon, because user program and the
syslog are in two different processes. Most context switches flush the
CPU pipeline and hence affecting efficiency.
4. Including a daemon process in mediating log affairs makes message
logging dominated by kernel's scheduling policy and number of running
processes in the system. Because CPU resources and CPU time are shared
by all the processes running in the system, the more running processes
in the system the less CPU time a single process (rsyslogd in our
case) can get to accomplish its job.
5. Because all running processes in the system share the same CPU, the
kernel must execute each process in turn (let each process get a
chance to be executed) such that CPU time are divided and distributed
among all running processes. The order of process execution is
determined by kernel's scheduling policy. The execution order could be
like the steps below:
(1) a user program writes a log message to the syslog socket.
(2) the kernel decides to execute two other processes in the
system first before executing the rsyslog daemon since they have
higher priority or due to the scheduling policy.
(3) the kernel than finally execute rsyslog daemon so that the log
message sent by user program at step (1) is received by the daemon
from syslog socket.
as more and more processes run in the system, the number of processes
get executed at step (2) could be increased. So the time of a log
message getting propagated from user program to rsyslog daemon becomes
longer and longer. And the latency to transfer a log message becomes
higher and higher. I think this kills the performance the most.
6. Log messages may need to stay in socket buffer for longer time due
the high latency which implies a bigger socket buffer is required
which implies a greater memory footprint. The case gets worse when
system loading is high and increase the risk of socket buffer overflow
and message loss.
>
> then you have the problem of what happens to the log. If you just write to
> disk, you throw away a tremendous amount of flexibility in determining what
> happens to the log.
>
> If you send the log through the syslog infrastructure, there are a lot of
> things that you can end up doing with it.
>
> You can:
>
> combine the logs from differnet programs into one file
>
> split the logs from one program into multiple files
>
> filter the logs
>
> send the logs to a remote machine
>
> put the logs into a database
>
> etc.
>
>
>
> so having your program write directly to disk is less work overall for the
> system to do, but it can be slower for your application, and it's far less
> flexible.
>
> David Lang
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
Justin Lee
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
