On Fri, 17 Jan 2014, Assaf Gordon wrote:
Hello,
On 01/17/2014 04:35 PM, David Lang wrote:
RELP will prevent logs from being lost in transit, but if the log
arrived on the server, but was in the queue when rsyslog was killed, and
rsyslog didn't have time to write it out, you can still loose things.
how did you kill the rsyslog receiver?
Hmmm I see.
I killed it with "kill", probably not the best way to do it, but if/when an
error happen, it wouldn't be "a nice way" either...
correct, but RELP is not protecting against that, it's protecting against
network failures
what does your config look like? (how much stuff are you doing with the
logs?)
My config files are exactly as I list below, nothing more.
I do not plan to add anything (unless there are good suggestions for this
issue).
It will be a separate 'rsyslogd' process, dedicated for application-level
logging.
do you have any idea what your queue stats looked like? (impstats)
No idea, I did not check those.
If you are having the logs written to a database or something like that,
it's not uncommon for it to take long enough to get the logs out that
rsyslog isn't able to do so at shutdown.
My server side logging is to a local file.
Thanks for your help!
Are there any settings that can mitigate such issues?
Yes there are, but the trouble is the performance hit. I have run rsyslog in a
configuration where there is no chance of loosing a log, and it's performance
slowed down to worse than 1/1000 of it's normal performance, and that was with a
_very_ expensive disk (PCI flash drive, $5k for 160G), with a more normal disk
things would be even worse.
The process of a message flowing through rsyslog is roughly:
1. the input module receives the message and puts it on the main queue (RELP
acks the message at this point)
2. an action worker thread marks the message as pending output
3. the action worker attempts to deliver the message (write it to the system's
disk cache)
4. if the output succeeds for all outputs, the action worker marks the message
as complete and removes it from the main queue.
note that if you use an action queue, you have a process that copies the message
from the main queue to the action queue with a process very similar to steps 2-4
and then the action worker works on the action queue instead of the main queue.
By default, these queues live in ram, and the writes to disk are allowed to be
buffered by the OS and will get to disk eventually, but not immediately.
you can configure rsyslog do do periodic fsync calls to force data to be written
to disk, and can do this as frequently as every message.
You can configure rsyslog to use a disk-only queue (not a disk-assisted queue)
and instead of keeping the queued logs in memory, rsyslog will write them to
disk. Just like the writes to a file, you can configure how paranoid rsyslog is
on making sure updates to these disk queues actually land on disk as opposed to
just being in the systems's disk cache. If you configure it to be super paranoid
and do a fsync after each queue update, you have no chance of loosing data
there.
But the result of this is that instead of these queue updates being updates to
shared memeory, with minor locking conflict, they end up being writes to disk +
fsync after each write and the performance goes through the basement.
There are some logs that are critical enough to get this sort of treatment, but
very few. For everything else you need to make a judgement of what sort of
failure conditions are acceptable to you.
Part of the reason that traditional syslog got the reputation of being unusably
slow is that it did fsyncs for every write to disk, and it did _no_ queueing.
rsyslog will be several times slower than traditional syslog if set to it's most
paranoid settings because it ends up doing quite a few writes to disk for every
message that it processes (but it does have RELP so you can pass messages from
system to system in this mode, traditional syslog couldn't).
now, with all of this said, let's get back to your situation and think about
what you are trying to defend against.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
