On Sat, 26 Jul 2014, Justin Lee wrote:
But in addition to disk writing, use of socket by syslog to transfer
local log messages introduces extra CPU time or memory to do the
following things:
1. To handle socket related stuff, such as copying log messages in and
out of socket buffer. Buffer copying introduces overhead.
2. We need a thread or daemon process (rsyslogd) to listen to and
receive log messages from the socket. It increases the memory
footprint of our log infrastructure because of the daemon process
itself.
3. At least one context switch is required to transfer a log message
from user program to rsyslog daemon, because user program and the
syslog are in two different processes. Most context switches flush the
CPU pipeline and hence affecting efficiency.
4. Including a daemon process in mediating log affairs makes message
logging dominated by kernel's scheduling policy and number of running
processes in the system. Because CPU resources and CPU time are shared
by all the processes running in the system, the more running processes
in the system the less CPU time a single process (rsyslogd in our
case) can get to accomplish its job.
5. Because all running processes in the system share the same CPU, the
kernel must execute each process in turn (let each process get a
chance to be executed) such that CPU time are divided and distributed
among all running processes. The order of process execution is
determined by kernel's scheduling policy. The execution order could be
like the steps below:
(1) a user program writes a log message to the syslog socket.
(2) the kernel decides to execute two other processes in the
system first before executing the rsyslog daemon since they have
higher priority or due to the scheduling policy.
that assumes that you don't give the rsyslog process a high enough priority,
which is a configuration error.
Process scheduling is not that simple to be controlled by merely
setting priorities or niceness value.
Since Linux allow more than one processes have the same priority,
rsyslogd could still be preempted by other processes with the same
priority even if the given priority is the highest one, as in 'Kernel
preemption on same priority tasks'
<http://stackoverflow.com/questions/17831970/kernel-preemption-on-same-priority-tasks>:
"in general case, if your processes have equal priorities, process
with less run-time will preempt another process on the next schedule()
invokation."
In addition, if the latency due to scheduling could be resolved by
configuring priority, the two projects D-Bus & Wayland should have
adopted it and no need to move code to the kernel or offload from
server to client processes. So in conclusion, giving rsyslogd high
priority doesn't resolve the issue.
Is there really an issue in practice though?
What is the worst-case delay in writing to /dev/log and being able to continue
executing?
Yes, the syslog standard says this is a blocking write and the app stops until
rsyslog reads the log message, but given multi-core systems, couldn't you give
rsyslog a high enough priority to always be running on one core so it doesn't
have to compete with other applications to run on that core?
Remember that if you are writing to disk, you have to wait for the disk
subsystem to get your data out, and even if you do buffered writes (letting the
kernel write your data out later), if there is too much writeback data pending,
the kernel can delay your app to slow down how much it's writing out.
Another approach:
What if you generated a UDP packet to localhost instead and let the kernel
buffer everything?
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
